The Complete vLLM Guide

This is part one of three, and it is written for someone who has never run a large language model on their own hardware. By the end you will be able to install vLLM, serve an open-weight model behind an HTTP endpoint that any OpenAI client can talk to, call it from curl and Python, run the same model offline from a script, size its memory so it starts reliably, and read the errors it throws when it does not. The guide is checked against vLLM 0.30.0, released in September 2026. vLLM ships a minor release roughly every two weeks and removes flags as it goes, so a lot of tutorials you will find online show options that no longer exist. Where that matters, this guide says so.

Each section ends with a Try it task. Do them. A server that loads a model onto a GPU teaches you more in one failed start than a page of reading, and the memory numbers in this guide only mean something once you have watched your own machine produce them.

What vLLM is, and the problem it solves

A large language model is a file of numbers (the weights) plus a program that, given some text, predicts the next piece of text one token at a time. A token is a chunk of a word, roughly three to four characters of English. To answer a question, the model predicts a token, appends it to the input, predicts the next, and repeats until it decides to stop. Producing a 300-token answer means running the model 300 times in sequence.

You can run that loop yourself with the Hugging Face transformers library, and for one user in a notebook it works fine. The trouble starts when several people use it at once. A GPU is an expensive machine that is only efficient when it has a lot of work to do in parallel. Serving one request at a time leaves most of it idle. Naive batching, where you wait for eight requests, run them together, and return them all when the slowest one finishes, wastes time on the short answers that finished early. And the model needs a large working memory for every request in flight, which a naive server reserves in big contiguous slabs sized for the worst case, so most of that memory sits empty.

vLLM is an inference and serving engine built to fix exactly those three problems. It is an open-source project that started at UC Berkeley and is now developed by a large community. You give it a model, it loads the weights onto your GPU or GPUs, and it serves many requests at once while keeping the hardware busy. It does this with two main ideas that you will meet in the next section: continuous batching and PagedAttention.

What came before it was a mix of approaches. You could write your own serving loop around transformers, which is flexible but slow under load. You could use a dedicated server such as Text Generation Inference, or a model-specific runtime. Or you could call a hosted API and never think about GPUs at all. vLLM sits in the middle: it is a general-purpose server for open-weight models that you run yourself, and it speaks the same HTTP dialect as the OpenAI API, so code written for a hosted API can be pointed at your own machine by changing one URL.

Why would you run a model yourself? Cost at volume is one reason. Control is another: you pick the model, the version, and the behaviour. A third reason is data residency. Many employers in the Gulf and in Egypt need prompts and documents to stay inside a particular country or a private network, and a model served from your own infrastructure never sends data to a third party. vLLM does not make that decision for you, but it makes the self-hosted option practical.

The diagram is the whole shape of the tool. Requests arrive over HTTP, a scheduler decides which requests share the next step on the GPU, and the GPU holds both the model and a pool of working memory. Everything else in this guide is detail about one of those four boxes.

The mental model: four ideas you must know

vLLM has a large surface, with more than two hundred command-line flags, but the behaviour you need to predict as a beginner comes from four ideas. If you hold these, almost every error message will make sense.

First, the KV cache. When a model reads your prompt, it computes for every token two vectors called a key and a value, at every layer. The next token's prediction needs to look back at all of them. Recomputing them for every new token would be hopelessly slow, so the engine stores them. That store is the KV cache. It grows by one token's worth of keys and values for every token the request has seen, and it lives in GPU memory. For most deployments the size of the KV cache, not the size of the weights, decides how many requests you can serve at once. Remember that sentence; it explains most memory errors.

Second, PagedAttention. Older servers reserved one big contiguous block of memory per request, big enough for the longest possible answer, and most of it went unused. vLLM instead cuts the KV cache into small fixed-size blocks and hands them out on demand, the way an operating system hands out pages of RAM. A request holds a list of blocks (a block table) that need not be next to each other. Nothing is reserved in advance, nothing is stranded between requests, and memory is returned the moment a request finishes. This is the idea the project is named for and the reason it fits many more concurrent requests on the same GPU.

Third, continuous batching. A server that batches whole requests waits for the slowest one. vLLM's scheduler instead decides, at every single token step, which requests run next. A request that finishes leaves immediately and a waiting one takes its place on the very next step. The GPU is therefore always working on a full batch. The scheduler works with a token budget per step (the flag --max-num-batched-tokens) and a cap on concurrent requests (--max-num-seqs).

Fourth, prefill and decode. Every request has two phases. Prefill reads your whole prompt in one go; it is heavy on computation and determines how long you wait for the first token (time to first token, or TTFT). Decode then produces the answer one token per step; it is limited by memory bandwidth and determines how fast tokens stream out. Long prompts are split into chunks so they do not block the decoding of other users, a feature called chunked prefill that is on by default. Another default is automatic prefix caching: if two requests start with the same text, such as the same system prompt, the engine reuses the KV blocks it already computed for it and skips that part of the prefill.

There are also a few nouns for the software itself. The LLM class is the Python interface for running a model inside your own script with no server, called offline inference. The command vllm serve starts an OpenAI-compatible server and is called online serving. Behind it, an API server process handles HTTP and tokenization, an engine core process runs the scheduler and manages the KV cache, and one worker process per GPU holds the weights and runs the model. You will see these names in logs. You do not need to manage them.

What hardware you need

The first question every beginner asks is whether they can try this on a laptop. The honest answer depends on the machine, and the official platform matrix is worth knowing before you spend an afternoon on a setup that cannot work.

The primary, best-supported platform is Linux with an NVIDIA GPU of compute capability 7.5 or higher. That covers the T4, RTX 20-series and newer, L4, A100, H100 and B200. AMD GPUs on Linux (the MI200, MI300 and MI350 families and recent Radeon cards) are supported through ROCm. Intel GPUs and Google TPUs have their own, more specialised paths. CPU-only Linux is supported, with prebuilt wheels, and is useful for learning the API with a tiny model, though it is slow.

Windows is not supported natively. The documentation is direct about this. The supported route is WSL2, the Windows Subsystem for Linux, where you install the NVIDIA driver on the Windows side and then follow the Linux steps inside an Ubuntu shell. Docker Desktop with the WSL2 backend and an NVIDIA GPU also works, since it runs the Linux image.

macOS on Apple silicon is experimental. vLLM runs on the CPU backend there and only from a source build, with 32-bit and 16-bit floating point. It is fine for learning the API with a tiny model and not a path to good performance. For Apple GPU acceleration the docs point to a community plugin called vllm-metal, which is separate from the core project. If your goal is simply to chat with a model on a MacBook, a tool such as Ollama or llama.cpp is a better fit; come back to vLLM when you have a Linux GPU box or a cloud instance.

How much GPU memory do you need? A rough rule: a model stored in 16-bit precision needs about two bytes per parameter for the weights, so a 7-billion-parameter model needs about 14 GB, plus room for the KV cache and some overhead. A 24 GB card runs a 7B or 8B model comfortably with a moderate context. A 0.6-billion-parameter model such as Qwen3-0.6B, which this guide uses for every example, needs little more than a gigabyte and a half for weights and runs on almost any NVIDIA card with a few GB free. That is why it is the official smoke-test model: you prove the whole pipeline works before you download a model that takes half an hour.

Installing vLLM and checking the setup

vLLM depends on one specific version of PyTorch, and its wheels contain compiled GPU kernels built against that exact version. That is why the install guidance is strict: use a clean virtual environment, and do not install or upgrade PyTorch yourself afterwards. The documented path uses uv, a fast Python package manager, and Python 3.12. The package metadata allows 3.10 to 3.14 and the docs mention 3.10 to 3.13, but 3.12 is what the docs' own example uses and what the AMD and Intel builds require, so teach yourself 3.12 and avoid the question.

On Linux with an NVIDIA GPU:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The --torch-backend=auto flag tells uv to look at your installed driver and pick the matching PyTorch build. The default vLLM wheel is built for CUDA 12.9. If the install fails with an index error, update uv itself with uv self update and retry. If you do not have uv, install it first from its official site, or use plain pip with the extra index URL documented on the installation page. In a course, pin the version so that your results match this guide:

uv pip install vllm==0.30.0 --torch-backend=auto

Do not install the CUDA Toolkit expecting it to be needed. You need an NVIDIA driver that is recent enough for the CUDA runtime in the wheel; the toolkit with the nvcc compiler is only for building vLLM from source.

Now verify, step by step, from the cheapest check to the most expensive:

nvidia-smi
python -c "import vllm; print(vllm.__version__)"
vllm --version
vllm collect-env

nvidia-smi shows the driver sees your GPU. The next two print the installed version, which should be 0.30.0. vllm collect-env prints a full report of your environment, including the PyTorch, CUDA and driver versions, and it is exactly what maintainers ask you to paste into a bug report, so it is worth running once to see what it contains.

If you are on Windows, open PowerShell and run wsl --install, restart, install the NVIDIA Windows driver (do not install a Linux driver inside WSL, since WSL passes the GPU through), open the Ubuntu shell, and follow the Linux steps above inside it. If a source build runs out of memory under WSL, export MAX_JOBS=1 slows the build but lowers the peak. On a CPU-only Linux machine, the project publishes CPU wheels and CPU Docker images; the CPU installation page gives the exact wheel URL, and you set VLLM_CPU_KVCACHE_SPACE to the number of gigabytes you want for the KV cache.

Two small habits belong here. First, gated models such as Meta's Llama need you to accept a licence on Hugging Face and then supply a token; export it as HF_TOKEN in the shell that runs vLLM. The model we use, Qwen3-0.6B, is not gated and needs no token. Second, vLLM collects anonymous usage statistics by default. If you or your employer want that off, set VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1, or create the file ~/.config/vllm/do_not_track.

Your first program: offline inference

The smallest working vLLM program does not need a server at all. It loads a model into your own Python process, generates text for a list of prompts, and exits. This is called offline inference and it is ideal for batch jobs, such as classifying ten thousand support tickets overnight, and for first experiments.

Save this as first.py:

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM(model="facebook/opt-125m")
    params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=40)
    outputs = llm.generate(["The capital of France is"], params)
    print(outputs[0].outputs[0].text)

Run it with python first.py. The first run downloads the model from the Hugging Face Hub into a cache under your home directory, so it takes a moment. You will see a long stream of log lines as vLLM loads the weights, compiles the model, captures CUDA graphs, and measures how much memory is left for the KV cache. Then a short continuation of the sentence prints. The model facebook/opt-125m is tiny and not instruction-tuned, so do not expect a clever answer; the point is that the pipeline works.

Read the code from the top. LLM(model=...) is the engine: constructing it is the slow step, because it is where the weights load and the GPU memory is planned. Construct it once and reuse it. SamplingParams holds the decoding settings for a request, covered below. llm.generate takes a list of prompts and returns a list of result objects in the same order. Each result has an outputs list (one entry per requested completion), and each entry has .text. You pass many prompts in a single call and vLLM batches them for you; looping over generate with one prompt at a time throws away the very thing the library is good at.

The if __name__ == "__main__": guard is not optional. vLLM starts worker processes, and on platforms where child processes re-import your script, an unguarded LLM(...) at the top of the file starts the engine recursively. The symptom is RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase, sometimes preceded by a warning that CUDA was previously initialized and the spawn method is being forced. Put everything that creates the LLM object inside the guard and the error disappears.

For an instruction-tuned model, you usually want to give it a conversation rather than raw text, because the model was trained on a specific chat format. The chat method applies the model's own chat template for you:

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM(model="Qwen/Qwen3-0.6B")
    messages = [
        {"role": "system", "content": "You answer in one short sentence."},
        {"role": "user", "content": "What is a GPU used for?"},
    ]
    out = llm.chat(messages, SamplingParams(temperature=0.7, max_tokens=200))
    print(out[0].outputs[0].text)

A chat template is a small recipe, shipped inside the model's tokenizer files, that turns a list of role-tagged messages into the exact string of special tokens the model was trained to expect. Using the wrong format makes a good model answer badly, which is why chat and the server's chat endpoint exist. Qwen3 is a model that can print its reasoning inside think tags before answering, so you may see some of that in the text; that is the model, not vLLM.

Your first server

For most real use you want vLLM as a service that other programs call. The command is vllm serve followed by the model name:

vllm serve Qwen/Qwen3-0.6B

The older form python -m vllm.entrypoints.openai.api_server was deprecated in 0.29; if a tutorial tells you to use it, use vllm serve instead and treat the rest of the tutorial with a little suspicion.

The server prints a lot at startup. Roughly, it downloads the model if it is not cached, loads the weights, compiles the model and captures CUDA graphs (a one-time cost that makes later steps faster), measures free memory, and sizes the KV cache. When it is ready, you will see lines saying the application startup is complete and that the server is listening on port 8000. Startup for a small model takes tens of seconds to a couple of minutes. For a large model it can take many minutes, which surprises people who expect a web server to be instant.

While it runs, open a second terminal and check three endpoints:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models
curl http://localhost:8000/version

/health returns an HTTP 200 with an empty body once the engine is ready, and is the check a load balancer or a Kubernetes probe would use. /v1/models lists the model names the server will answer to, which is important in the next section. /version returns the vLLM version.

Notice what the server does not do. By default it binds all network interfaces, since --host defaults to nothing, which means any machine that can reach yours can reach port 8000. On a laptop on a café network, or a cloud machine with an open security group, that is an unauthenticated model endpoint open to the world. For anything beyond a throwaway test, set the host explicitly:

vllm serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 8000

With 127.0.0.1 only processes on the same machine can connect. We return to authentication and safe exposure later.

Talking to the server

The server speaks the OpenAI API dialect, so you can use the official openai Python package, plain curl, or any tool that accepts a custom base URL. Start with curl, because it shows exactly what crosses the wire:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

The reply is a JSON object. The answer lives at choices[0].message.content. There is also a usage object with prompt_tokens, completion_tokens and total_tokens, which is how you learn what a request cost in tokens, and a finish_reason: stop means the model decided it was done, while length means it hit your max_tokens limit and was cut off.

The same call from Python with the OpenAI client, installed alongside vLLM:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Tell me a joke about GPUs."}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Two details trip people up. The base_url must end in /v1, because that is the path prefix of the OpenAI-style routes. And the api_key is required by the client library even though the server is not checking one yet, so any placeholder string works.

The model field must match a name the server knows. By default that is the exact model path you gave vllm serve. If you send a different name, such as gpt-4, the server replies with HTTP 404 and a message that the model does not exist. You can choose a friendlier public name with --served-model-name:

vllm serve Qwen/Qwen3-0.6B --served-model-name tiny-chat --host 127.0.0.1

Clients now send "model": "tiny-chat". This is useful because it lets you swap the model behind the name without touching client code, which is the same trick a hosted API plays.

To stream tokens as they are produced, which is what makes chat interfaces feel fast, add stream=True and iterate over the chunks:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

stream = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Explain KV cache in two sentences."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()

Besides /v1/chat/completions, the server also offers /v1/completions, the older raw-text endpoint that takes a plain prompt and applies no chat template, and /v1/embeddings for models built to produce embeddings. For a chat model, use the chat endpoint. vLLM also has convenience clients in the terminal: with a server running, vllm chat --quick "hi" sends a one-shot message to it.

Because the interface is standard, the rest of the ecosystem plugs in. Frameworks that accept an OpenAI base URL work unchanged, and a gateway such as LiteLLM can sit in front of one or many vLLM servers to give you keys, budgets and routing. If you have used the hosted API described in the OpenAI API guide, the only changes are the base URL and the model name.

Controlling the output: sampling parameters

The model does not output a word. It outputs a probability for every token in its vocabulary, and a sampling step chooses one. The settings that govern the choice are the main way you control how an answer reads. In the offline API they live in SamplingParams; through the server you pass them in the request body, with the vLLM-specific ones inside extra_body when you use the OpenAI client.

Temperature reshapes the probabilities. Near zero, the model almost always picks the single most likely token, so answers are focused and repeatable. At 1.0 it samples more freely, and above that answers get erratic. top_p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to that fraction, so top_p=0.9 ignores the long tail of unlikely words. top_k keeps only the k most likely tokens; vLLM accepts it, though it is not part of the standard OpenAI API, so with the OpenAI client you send it as extra_body={"top_k": 20}. max_tokens caps the length of the answer, and stop gives strings that end generation when produced. n asks for several independent completions of one prompt. seed makes sampling reproducible for a given setup.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

r = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Name three colours."}],
    temperature=0.2,
    top_p=0.9,
    max_tokens=60,
    extra_body={"top_k": 20},
)
print(r.choices[0].message.content)

A subtle and important default: if you do not send a setting, vLLM may fill it in from the model's own generation_config.json. Model authors ship recommended values for temperature, top_p and similar, and since version 0.8 vLLM applies them as the server's defaults. This is helpful, since you get the author's intended behaviour, but it explains the confusing report "I changed nothing and the answers changed after an upgrade" or "the same prompt behaves differently from another server". If you want vLLM's own neutral defaults instead, start the server with --generation-config vllm. In offline code the equivalent is LLM(model=..., generation_config="vllm"). Being explicit by always sending the parameters you care about is the safest habit.

The flags you will actually use

vllm serve accepts hundreds of flags, which is intimidating. A beginner needs about ten. The server also lets you explore the rest yourself: vllm serve --help=listgroup lists the flag groups, vllm serve --help=<GroupName> prints one group, and vllm serve --help=<flag> explains a single flag.

Here is a realistic first configuration, and then each flag explained:

vllm serve Qwen/Qwen3-0.6B \
  --host 127.0.0.1 --port 8000 \
  --served-model-name tiny-chat \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --api-key "$VLLM_API_KEY"

--host and --port choose where the server listens, as above. --served-model-name sets the name clients use. --api-key makes the server require an Authorization: Bearer <key> header on its main API routes; you can also set the VLLM_API_KEY environment variable instead of putting the key on the command line, which keeps it out of the process list that other users on a shared machine can see.

--max-model-len is the maximum number of tokens in one request, counting both the prompt and the answer. If you omit it, vLLM uses the context length from the model's configuration, which for modern models can be tens of thousands of tokens or more. A long maximum is not free: the engine must be able to fit at least one full-length request in the KV cache. Setting it to what you actually need, such as 8192 here, is the single most common fix for memory startup errors. The flag accepts friendly forms like 8k.

--gpu-memory-utilization is the fraction of the GPU's total memory that this vLLM instance may use for weights, working space and KV cache combined. The default is 0.92. Many older guides say 0.9, which was the previous default. It is a cap on the instance, not "extra" memory on top of the model, and it is a fraction of the total, not of what is currently free. If something else already holds memory on the GPU, a high value can fail at startup. Whatever is left after the weights and working memory becomes the KV cache, so raising the number gives you more concurrency and lowering it gives you headroom for other programs.

--dtype sets the numeric precision of the weights. The default, auto, uses the precision recorded in the model, normally bfloat16. Older GPUs such as the T4 (compute capability 7.5) do not support bfloat16; there you pass --dtype half. --trust-remote-code allows a model that ships its own Python code to run that code. It is off by default for a good reason: a model repository can contain arbitrary code, so only enable it for repositories you trust. --tensor-parallel-size splits one model across several GPUs of one machine when it does not fit on one; it is a topic for the next level. --download-dir and --revision control where weights are cached and which version of the repository is used.

--enforce-eager turns off CUDA graph capture. Startup is faster and memory use is slightly lower, at the cost of slower generation. It is useful for debugging and quick experiments, not for production.

If you find yourself repeating the same long command, move it into a YAML file whose keys are the flag names without the dashes, and run vllm serve --config config.yaml. Anything you also type on the command line wins over the file.

Memory, the KV cache and reading the startup log

Because memory causes most beginner failures, it is worth understanding how vLLM budgets it. At startup the engine does three things in order. It loads the weights. It runs a dummy pass through the model to measure how much temporary working memory the computation needs. Then it takes the budget (the total memory times --gpu-memory-utilization), subtracts the weights and the working memory, and gives everything left to the KV cache, which it carves into blocks.

The startup log reports this. You will see a line giving the memory the model weights took, and lines reporting the GPU KV cache size in tokens and the maximum concurrency for requests of your maximum length. Read the KV cache size as the total number of tokens that can be in flight across all requests at once. If it says the cache holds 50,000 tokens and each request uses about 1,000 tokens, you can serve around fifty requests together. If requests are long, the number of simultaneous requests drops; if they are short, it rises. This is the quantity you tune.

A tiny numeric example makes it concrete. Imagine a GPU with 24 GB, a utilization of 0.9 (21.6 GB budget), weights of 16 GB and working memory of 2 GB. That leaves about 3.6 GB for the KV cache. Raise utilization to 0.95 and the budget grows to 22.8 GB, so the KV cache grows to 4.8 GB, a third more room for requests. Cut --max-model-len and nothing about that pool changes, but you stop the engine rejecting startup because it could not fit one huge request. The numbers here are illustrative, not measurements from a specific model.

What happens when the cache fills mid-run? The scheduler preempts a request: it evicts its blocks and later recomputes them when there is room again. In the current engine this is by recomputation, not by swapping memory to the CPU, so the old --swap-space flag no longer exists. Preemption is not an error, but it costs speed and shows up as a warning. The cure is more KV memory: raise the utilization, shorten the maximum length, quantize the model, or use more GPUs.

Two other things appear in the log and are worth recognising. One is the compilation and CUDA graph capture step, which can take a while the first time and is cached afterwards under ~/.cache/vllm, the directory named by VLLM_CACHE_ROOT. The other is the periodic stats line that reports throughput and the number of running and waiting requests, a handy live view of load.

Common errors and how to read them

Most vLLM failures fall into a small set. When a start fails, the last line is often a generic wrapper, RuntimeError: Engine core initialization failed. See root cause above. Read that literally: scroll up a few lines. The real exception is above it, and that is the one to search for.

Not enough memory for one full request. The message begins To serve at least one request with the model's max seq len (131072), (16.00 GiB KV cache is needed, which is larger than the available KV cache memory (9.23 GiB). Based on the available memory, the estimated maximum model length is 75584. Your numbers will differ. It means the model's default context is longer than the leftover memory can hold. It even tells you a length that fits. Pass that as --max-model-len, or a smaller figure, raise --gpu-memory-utilization, or choose a smaller or quantized model.

No room for the cache at all. No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine. The weights and working memory already exceed the budget. A bigger GPU, more GPUs, or a smaller model are the real fixes; raising utilization helps only if you were just under the limit.

The GPU is already in use. Free memory on device cuda:0 (10.5/79.2 GiB) on startup is less than desired GPU memory utilization (0.92, 72.9 GiB). Another process holds memory: a second vLLM, a notebook, or a zombie from a crashed run. Run nvidia-smi, find the process, and stop it, or lower the utilization. Two instances on one GPU need roughly 0.45 each.

Wrong model name, HTTP 404. The model `gpt-4` does not exist. The model in the request does not match the served name. Check /v1/models.

Prompt too long, HTTP 400. This model's maximum context length is 4096 tokens. However, you requested 1000 output tokens and your prompt contains 3500 input tokens, for a total of 4500 tokens. Prompt plus requested output must fit within the maximum length. Shorten one of them or serve with a larger --max-model-len if memory allows.

bfloat16 on an old GPU. Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5. You can use float16 instead by explicitly setting the dtype flag. Add --dtype half.

Custom code needed. Failed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting trust_remote_code=True. Either upgrade the libraries or add --trust-remote-code, and only after reading what the repository's code does.

Unguarded offline script. The bootstrapping RuntimeError described earlier. Add the __main__ guard.

Device cannot be found. RuntimeError: Failed to infer device type usually means the wrong wheel (a CPU build on a GPU machine, or the reverse) or no visible GPU. Run VLLM_LOGGING_LEVEL=DEBUG vllm serve ... for more detail.

A hang with no error. Usually a download or a slow disk. Pre-download the model with the Hugging Face CLI, and keep weights on a local disk rather than a network share.

For anything unclear, VLLM_LOGGING_LEVEL=DEBUG gives much more output, and vllm collect-env gives the facts a maintainer will ask for.

Running vLLM in Docker

Once the server works on your machine, the natural next step is a container, so the environment is the same everywhere. If Docker is new to you, read the Docker guide first. The project publishes an image named vllm/vllm-openai, and everything you put after the image name is passed to vllm serve:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:v0.30.0 \
  --model Qwen/Qwen3-0.6B

Each piece has a reason. --runtime nvidia --gpus all gives the container access to the GPU through the NVIDIA container toolkit, which must be installed on the host. The -v mount shares your Hugging Face cache so the model downloads once, not at every start. HF_TOKEN passes your token for gated models. -p 8000:8000 publishes the port. --ipc=host (or a generous --shm-size) matters because PyTorch and the GPU communication library use shared memory, and Docker's small default makes multi-GPU runs fail with obscure errors. Finally, a pinned tag such as v0.30.0 is better than latest, because latest will move under you and a restart might quietly give you a new version with removed flags.

Add -v vllm-cache:/root/.cache/vllm to persist the compile cache so restarts do not recompile. If you publish the port to a network, remember that the container binds all interfaces; publish as -p 127.0.0.1:8000:8000 to keep it local, and place a gateway in front for anything shared. For the Kubernetes version of this deployment, see the Kubernetes guide, and note the official warning to avoid naming a Kubernetes Service vllm, since Kubernetes injects environment variables named VLLM_* that collide with vLLM's own settings.

Putting it all together

Here is one small project that uses everything above: a local, key-protected summarisation service and a client that summarises text files. The goal is a repeatable setup you can hand to a colleague.

First write a configuration file, so the command line stays short:

model: Qwen/Qwen3-0.6B
host: "127.0.0.1"
port: 8000
served-model-name: summariser
max-model-len: 8192
gpu-memory-utilization: 0.85

Start it with the key in the environment, not in the command:

export VLLM_API_KEY="change-me-for-real-use"
vllm serve --config config.yaml

Then write the client. It reads the key from the same environment variable, so no secret is in the code:

import os
import sys
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["VLLM_API_KEY"],
    base_url="http://127.0.0.1:8000/v1",
)

def summarise(text: str) -> str:
    r = client.chat.completions.create(
        model="summariser",
        messages=[
            {"role": "system", "content": "Summarise the user's text in three bullet points."},
            {"role": "user", "content": text},
        ],
        temperature=0.2,
        max_tokens=200,
    )
    return r.choices[0].message.content

if __name__ == "__main__":
    for path in sys.argv[1:]:
        with open(path, encoding="utf-8") as f:
            print(f"== {path}")
            print(summarise(f.read()))

Run python summarise.py notes.txt report.txt. Because the system prompt is the same for every call, prefix caching reuses its blocks after the first request, a small free speed-up. Check the server is healthy with curl http://127.0.0.1:8000/health before you start a batch. Note that the health route is not behind the key, which is a hint of something bigger: the API key only protects the main API routes, not every endpoint, so a bare vLLM server is not a safe thing to expose to the internet. Put it behind a reverse proxy that authenticates callers and only forwards the paths you intend.

Finally, if you want a single place for monitoring, the server exposes Prometheus metrics at /metrics, including running and waiting request counts and KV cache usage. Scrape them with Prometheus and chart them in Grafana. Like the health route, this endpoint is unauthenticated, so keep it on a private network.

What you can now do, and what comes next

You can now explain what vLLM is for and why it fits many more requests on one GPU than a naive loop: PagedAttention for memory, continuous batching for scheduling, prefix caching for repeated prompts. You can install a pinned version into a clean environment and verify it. You can run a model offline from a guarded script, and serve one with vllm serve, call it from curl and the OpenAI client, stream the answer, set sampling parameters, and understand where the unset defaults come from. You can budget memory with --max-model-len and --gpu-memory-utilization, read the startup log, run the same thing in Docker, and recognise the most common errors.

Some habits to carry forward: pin versions, bind to localhost until you have a gateway, size the context to your real needs, and read the error above the error.

The mid-level guide takes the same tool into real deployments: configuration files, tool calling, structured outputs, LoRA adapters, quantization, tuning the scheduler and benchmarking with vllm bench. The senior guide covers multi-GPU and multi-node scaling, security and the trust model, upgrades, cost and running vLLM as a platform. Neighbouring guides worth reading are Text Generation Inference and SGLang as alternative servers, LiteLLM as a gateway, KServe for running models on Kubernetes, and Langfuse for tracing the applications that call your model.

Sources

• vLLM documentation home
• Quickstart
• GPU installation
• CPU installation
• OpenAI-compatible server
• CLI reference
• Engine arguments
• Conserving memory
• Architecture overview
• Automatic prefix caching
• Docker deployment
• Troubleshooting
• Security
• Production metrics
• Release notes for v0.30.0

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about vLLM. Mid-level reviews this plus its own material; Senior reviews all three. The facts here are for vLLM 0.30.0. If an interviewer quotes a flag you do not recognise from an older blog post, it may well have been removed, and saying so calmly is a good answer.

The thirty-second answer

vLLM is an open-source inference and serving engine for large language models. It keeps the GPU busy by batching requests continuously, one token step at a time, and it stores each request's attention keys and values in small fixed-size blocks instead of big reserved slabs, which is called PagedAttention, so far more requests fit on the same GPU. It exposes an OpenAI-compatible HTTP server, so existing client code works by changing the base URL.

Then the line that shows you have run it: "memory is the thing that goes wrong: the KV cache, not the weights, usually decides how many requests I can serve, and I tune it with the maximum model length and the GPU memory utilization."

The model

Two sentences glue the table together. A request arrives, is tokenized, and waits in the scheduler. Each step the scheduler packs as many running and waiting requests as the token budget allows, the GPU produces one more token for each, and finished requests free their blocks at once.

Commands to have ready

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm==0.30.0 --torch-backend=auto

vllm --version
vllm collect-env
nvidia-smi

vllm serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 8000
vllm serve Qwen/Qwen3-0.6B --max-model-len 8192 --gpu-memory-utilization 0.85
vllm serve Qwen/Qwen3-0.6B --served-model-name tiny-chat --api-key "$VLLM_API_KEY"
vllm serve --config config.yaml

curl http://localhost:8000/health
curl http://localhost:8000/v1/models
vllm chat --quick "hi"
vllm serve --help=listgroup

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM(model="Qwen/Qwen3-0.6B")
    out = llm.generate(["The capital of France is"], SamplingParams(temperature=0.8, max_tokens=20))
    print(out[0].outputs[0].text)

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
r = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Hi"}],
)
print(r.choices[0].message.content)

Be able to say, without looking, what each of these does and in what order you would run them on a new machine: driver check, install, version check, serve, health check, models check, first request.

Classic traps and why they happen

The server will not start with a message about the model's max seq len. The model's default context length is larger than the memory left after the weights, so one full-length request cannot fit. vLLM refuses rather than failing later. The message prints an estimate of a length that does fit; pass it as --max-model-len.

"Free memory on device is less than desired GPU memory utilization." --gpu-memory-utilization is a fraction of the total memory per instance, and something else, often a previous run you forgot, already holds some. Check nvidia-smi.

The offline script prints a bootstrapping RuntimeError. The LLM(...) call was not under if __name__ == "__main__":. vLLM spawns worker processes, and an unguarded script re-runs itself in each.

HTTP 404 "The model does not exist". The model field does not match the served name. The default is the exact string you passed to vllm serve; --served-model-name changes it.

HTTP 400 about maximum context length. Prompt tokens plus requested output tokens exceed --max-model-len. This is a request problem, not a server fault.

Importing vLLM fails with undefined symbols after "fixing" a dependency. Someone installed a different PyTorch. vLLM wheels are built against one specific version; rebuild the environment cleanly.

The same prompt answers differently on a new version or a different server. Sampling defaults come from the model's generation_config.json. --generation-config vllm restores vLLM's own defaults. The honest answer to "is the output deterministic" is: nearly, with temperature zero and a seed on one setup, but not guaranteed across hardware or versions.

The endpoint is reachable by strangers. The default host binds every interface, and --api-key protects only the main API routes. Bind to localhost or place a gateway that authenticates and allowlists paths in front.

Common interview questions

What problem does vLLM solve? Serving language models efficiently under concurrent load. A naive loop serves one request at a time and leaves the GPU idle; static batching waits for the slowest request; contiguous memory reservation wastes most of the KV cache. vLLM answers with continuous batching and PagedAttention, and gains more from prefix caching and chunked prefill.

Explain PagedAttention to someone who knows operating systems. It is virtual memory for the KV cache. The cache is divided into fixed-size blocks; each request has a block table that maps its logical token positions to physical blocks that need not be contiguous. Blocks are allocated on demand and freed when the request finishes, so there is little fragmentation and no up-front worst-case reservation.

What is the KV cache and why does it matter more than model size at scale? It stores the keys and values for every token of every active request so the model does not recompute them at each step. It grows with the number of tokens in flight. Once the weights are loaded, the remaining memory becomes the KV cache, so its size caps concurrency. A model that fits is not enough: it must fit and leave room for the traffic.

What does continuous batching change? The scheduler decides at every token step which requests run. Finished requests leave immediately and waiting ones join, so the batch stays full and short requests are not held hostage by long ones.

What is the difference between prefill and decode, and why do you care? Prefill processes the prompt at once and is compute-bound; it determines time to first token. Decode generates one token per step and is memory-bandwidth-bound; it determines inter-token speed. Long prompts hurt first-token time and can disturb other users' decoding, which chunked prefill mitigates.

How do you run a model with vLLM, offline and online? Offline: construct LLM(model=...) once inside a main guard, then call generate or chat with a list of inputs and a SamplingParams. Online: run vllm serve <model> and call the OpenAI-style endpoints on port 8000, using /v1/chat/completions for chat models.

How do you point existing OpenAI code at vLLM? Set the client's base URL to http://host:8000/v1, use any placeholder or the real key if --api-key is set, and send the served model name as model. vLLM-specific parameters such as top_k go in extra_body.

What does --max-model-len do and how do you choose it? It caps prompt plus output tokens per request. By default it is taken from the model's configuration, which may be far larger than you need. The engine must fit at least one such request in the KV cache. Choose the longest request you truly expect; a smaller value frees memory for concurrency.

What does --gpu-memory-utilization mean, and what is its default? The fraction of total GPU memory this instance may use for weights, working memory and KV cache. The default is 0.92 in current versions, 0.9 in older material. It is a per-instance cap; two instances on one GPU need to split it.

Your server starts but throughput is poor and logs mention preemption. What is happening? The KV cache is filling, so requests are evicted and recomputed. It is a warning, not a crash. Increase the KV memory by raising utilization, shortening the maximum length, quantizing, or adding GPUs, or reduce concurrency.

What is automatic prefix caching, and when does it help? It reuses KV blocks for requests with the same starting tokens, such as a long shared system prompt. It only speeds prefill, never decode, so it helps most with long repeated prefixes. It is on by default.

Which platforms does vLLM support? Linux with NVIDIA GPUs of compute capability 7.5 or higher is the primary target; AMD through ROCm, Intel, TPU and CPU are available. Windows is only through WSL2. macOS on Apple silicon is experimental, CPU-only, and from source.

How do you check the install is healthy? nvidia-smi for the driver, vllm --version for the version, vllm collect-env for the full report, then serve a small model and check /health and /v1/models.

Why would you choose vLLM over a hosted API? Control of the model and version, predictable cost at high volume, and keeping prompts inside your network or country. Against that: you run GPUs, patch the software, and own availability.

Is the --api-key flag enough to secure the server? No. It guards the main API routes but not every path. Bind to a private interface and put a reverse proxy or gateway in front that authenticates and allowlists endpoints. Pass the key through the environment rather than the command line.

What changed recently that old tutorials get wrong? The older V0 engine and its toggle are gone. python -m vllm.entrypoints.openai.api_server is deprecated in favour of vllm serve. The --swap-space flag no longer exists. Request logging is off unless you opt in with --enable-log-requests. The structured-output fields were renamed. Citing two of these accurately shows you work with the current tool.

How would you run it in a container? Use the vllm/vllm-openai image with a pinned tag, --runtime nvidia --gpus all, --ipc=host or a shared-memory size, a volume for the Hugging Face cache, -p 8000:8000, and pass the server flags after the image name.

What is the relation between vLLM and tools like Ollama or llama.cpp? Ollama and llama.cpp target easy local use, often on laptops and CPUs. vLLM targets high-throughput serving on server GPUs. You may prototype in one and deploy in the other.

Questions to expect about your own experience

Interviewers at the beginner level often ask about what you actually did. Prepare short, true answers to these. Which model did you serve and on what GPU? What was the KV cache size reported at startup, and how did it change when you moved the utilization? Which error did you hit first, and how did you find the real cause? What did you change to make a request fail on purpose, and what did the 400 or 404 say? Did you try it in Docker and what did the shared-memory flag do?

A strong answer names numbers and error text. A weak one says "I followed a tutorial and it worked". You do not need production experience. You need to have broken it on purpose and fixed it, which the Try it tasks in the detailed track exist to give you.

What a strong answer covers that a weak one misses

A weak answer to "what is vLLM" says "a faster way to run LLMs". A strong answer names the bottleneck it attacks, which is GPU memory management and scheduling, and names the mechanism: PagedAttention and continuous batching.

A weak answer to a startup failure says "I would increase the GPU memory". A strong one reads the message, notices it is about the maximum model length versus KV cache, and reaches for --max-model-len first, utilization second, a smaller or quantized model third.

A weak answer to "is it secure" says "it has an API key". A strong one knows the key does not cover everything, and describes a gateway.

A weak answer to "how do I make output reproducible" says "set temperature to zero". A strong one adds a fixed seed, explains the model's generation config supplies defaults, and admits that exact reproducibility across hardware is not promised.

Sixty-second self-test

Answer out loud, without scrolling up.

1. What are the two big ideas in vLLM, in one sentence each?
2. What is the KV cache, and why does its size set your concurrency?
3. Name the two phases of a request and which one determines time to first token.
4. What is the command to start a server, and which port does it listen on by default?
5. A client gets a 404 saying the model does not exist. What do you check?
6. You see "To serve at least one request with the model's max seq len ...". Which flag do you reach for first?
7. What is the default value of --gpu-memory-utilization, and is it a fraction of free or total memory?
8. Why must an offline script use a main guard?
9. Why is --api-key alone not enough to expose a server publicly?
10. Which platforms are fully supported, and what is the route for Windows?

If you stumbled on more than two, reread the matching section of the detailed track and redo its Try it. The related guides on Docker, LiteLLM and Kubernetes cover the neighbours an interviewer is likely to ask about next.

Follow-up questions that probe depth

Interviewers like to pull one thread. Here are the follow-ups that most often come after the answers above, with the shape of a good reply.

"You said the KV cache limits concurrency. How would you estimate how many users you can serve?" Read the KV cache size in tokens from the startup log. Divide by the typical total length of a request, prompt plus answer. That gives a rough ceiling on simultaneous requests. It is a ceiling, not a promise, because the scheduler also caps concurrent requests with --max-num-seqs and spends a token budget per step. Then say you would confirm with a benchmark rather than trust the arithmetic. The command vllm bench serve exists for that.

"Why is time to first token sometimes long even on a fast GPU?" Either the prompt is long, so prefill takes time, or the request waited in the queue because the cache or the concurrency cap was full. Repeating a long prefix helps through prefix caching. A queue means you are short of capacity.

"What would you do if the server must stay inside a country?" Run it in a region or data centre that satisfies the rule, keep model weights local, avoid sending prompts to external services, and keep the endpoint on a private network behind a gateway. vLLM itself sends only optional anonymous usage statistics, which you can disable with VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1.

"A colleague says they set the temperature in the request but the output still feels different from the hosted API. Why?" Parameters they did not set come from the served model's generation config, and the hosted service may use different defaults, a different model version, or a different system prompt. Compare like with like by sending every parameter explicitly.

"When would you not use vLLM?" For a single user chatting on a laptop, a lighter tool such as Ollama or llama.cpp is simpler. For a workload that needs a model vLLM does not support on your hardware, pick another runtime. Say it plainly: choosing the right tool is part of the job, and no interviewer expects loyalty to one project.

"How do you know a request was cut off?" The response's finish_reason is length when max_tokens or the context limit stopped it, and stop when the model finished by itself. A beginner who ignores this ships truncated answers.

Part one of three. Almost every beginner problem with vLLM comes from one of four misunderstandings: memory is a budget with a fixed size, the model's own defaults quietly fill in what you did not set, the server is wider open than you assume, and vLLM changes quickly so old advice rots. Start with the table, then the longer entries, then the habits and checklist. Everything here is checked against vLLM 0.30.0.

Common errors at this level

Mistakes in detail

Asking for the model's full context when you need a fraction

Symptom. You start a modern model and it refuses with a message about the model's max seq len, saying a certain number of gibibytes of KV cache is needed but only a smaller number is available.

Cause. Many current models advertise very long context windows. Unless told otherwise, vLLM promises to be able to serve a single request of that full length, and it checks at startup that the KV cache is large enough. It is not, because you have a modest GPU.

Fix. The message includes an estimated maximum length that fits. Start with that, or with the length you need:

vllm serve Qwen/Qwen3-0.6B --max-model-len 8192

Then, if you truly need more, consider a quantized model or a bigger GPU rather than just fighting the flag. Shrinking the length does not lose anything you were using; it only removes the promise you were never going to cash.

Treating gpu-memory-utilization as "extra" memory

Symptom. You set --gpu-memory-utilization 0.9 expecting the model to use 90 per cent of what is free, then see a failure complaining that free memory is below the desired amount. Or you start a second server on the same GPU and the second one dies.

Cause. The value is a fraction of the GPU's total memory that one instance may use for everything: weights, working memory and KV cache. Two instances at 0.9 each cannot both fit. A leftover process from an earlier crash still holds its share.

Fix. Look with nvidia-smi before you start anything. Stop stray processes. To share a GPU, give each instance a share that adds up below one, for example roughly 0.45 each. For a single instance, leave a little headroom if a desktop session or notebook is also using the card.

Reading only the last error line

Symptom. The log ends with RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {...} and you search for that line and find nothing helpful.

Cause. That line is a wrapper added by the parent process when the engine process died. It never contains the reason.

Fix. Scroll upward. The real exception, such as a download failure, an authentication error for a gated model, an out-of-memory message, or an unsupported model architecture, is usually one to three blocks above. Copy that one into your search, not the wrapper. For gated models, make sure you accepted the licence on Hugging Face and exported HF_TOKEN.

Forgetting the main guard in offline scripts

Symptom. A script that worked as a snippet prints a long traceback ending with the message about starting a new process before the current process has finished its bootstrapping phase, sometimes after a warning about CUDA already being initialized.

Cause. vLLM starts worker processes. If your file creates the engine at import time, each child re-imports the file and tries to create another engine.

Fix.

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM(model="facebook/opt-125m")
    print(llm.generate(["Hello"], SamplingParams(max_tokens=10))[0].outputs[0].text)

The same applies inside notebooks and test files that import your module: keep engine creation in a function you call from the guarded block, not at the top level.

Calling generate in a loop with one prompt

Symptom. The offline job works but is much slower than the benchmarks you read about, and the GPU shows low utilization.

Cause. Continuous batching needs concurrent work. A loop that calls generate for one prompt at a time gives the scheduler one request at once.

Fix. Pass the whole list of prompts in a single call and let vLLM schedule them:

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM(model="Qwen/Qwen3-0.6B")
    prompts = [f"Write a one-line slogan for product {i}." for i in range(100)]
    outputs = llm.generate(prompts, SamplingParams(temperature=0.7, max_tokens=30))
    for o in outputs:
        print(o.outputs[0].text.strip())

For the server, the equivalent is to send many requests concurrently from the client, using threads or an async client, instead of waiting for each to finish before sending the next.

Sending a raw prompt to a chat model

Symptom. The answers are strangely flat, repeat the question, or ramble, though the model is known to be good.

Cause. Instruction-tuned models expect a particular chat format. The raw completions endpoint, and llm.generate with a plain string, apply no template, so the model sees text it was never trained to answer.

Fix. Use /v1/chat/completions on the server, or llm.chat(messages, ...) offline. Those apply the model's chat template. If a model has no template in its tokenizer files, provide one with --chat-template path/to/template.jinja.

Changing nothing and getting different answers

Symptom. After an upgrade or when moving to a new server, the same request produces different style or length.

Cause. vLLM takes default sampling settings from the model's generation_config.json. Different models, or a changed file, mean different defaults. Your request, which did not set temperature, silently inherited them.

Fix. Send the parameters you care about in every request. If you want vLLM's neutral defaults, start with --generation-config vllm. Record the model revision you serve; --revision pins it.

Copying flags from an old tutorial

Symptom. An unknown-argument error for a flag in a blog post, or an environment variable that does nothing.

Cause. vLLM releases about every two weeks and removes things on minor versions. Examples now gone or replaced: the V0 engine and its switch, the attention-backend environment variable, --swap-space, and the old guided-decoding request fields. The python -m vllm.entrypoints.openai.api_server launcher is deprecated in favour of vllm serve.

Fix. Check the flag against the installed version with vllm serve --help=<flag>, and against the official docs for your version. Pin your version so that the docs you read match the software you run.

Exposing the server as it comes

Symptom. Nothing visibly breaks, until someone else uses your GPU, or a scan finds an open model endpoint.

Cause. Without --host, the server listens on every network interface. And --api-key guards the main API routes, not every endpoint.

Fix. While learning, --host 127.0.0.1. Later, a reverse proxy that authenticates and forwards only the paths you want. Keep /metrics on a private network. Never run with the development-mode switch outside a lab.

Putting the API key on the command line

Symptom. Your key appears in shell history and in the process list.

Cause. --api-key mysecret is part of the command line, which other users on a shared machine can read with a process listing.

Fix. Export VLLM_API_KEY in the environment, or inject it from a secret store, and leave the flag off. In Docker or Kubernetes, use the platform's secret mechanism rather than baking it into an image.

Re-downloading and recompiling on every container start

Symptom. Each docker run takes many minutes before the server is ready.

Cause. The container's filesystem is thrown away with the container. Weights and compile artifacts go with it.

Fix. Mount the Hugging Face cache and the vLLM cache on volumes:

docker run --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:v0.30.0 --model Qwen/Qwen3-0.6B

Habits worth adopting on day one

Pin the version. Install vllm==0.30.0, record it, and upgrade on purpose. When something differs from this guide, the first question is which version you are on.

Use a dedicated environment. One virtual environment per project, created with uv, with nothing installed afterwards that touches PyTorch. If it ever behaves strangely, recreate it rather than repairing it; that takes a minute.

Begin with a tiny model. Prove the whole path, including the client, with Qwen3-0.6B before you download a model that takes half an hour and a lot of memory. When the small one works, any failure with the big one is about size, not setup.

Size the context to your use. Choose --max-model-len from your real prompts. Measure the longest prompt and add the longest answer you allow, then round up. It is the cheapest tuning you will ever do.

Read the startup log once, fully. Find the weights memory, the KV cache size and the concurrency estimate. Knowing what normal looks like makes abnormal obvious.

Check nvidia-smi before and after. Before you start, so you know what else holds memory; after you stop, to confirm the memory was released.

Send parameters explicitly. Temperature, top_p and max_tokens belong in your code, not in the model's hidden defaults.

Name your model. Use --served-model-name so clients do not depend on a path, and so you can change the model behind the name later.

Treat the log as the first debugger. VLLM_LOGGING_LEVEL=DEBUG costs nothing and shows far more. Keep vllm collect-env output for bug reports.

Keep secrets out of arguments. Use environment variables or a secret store.

Bind narrowly. Localhost until a gateway exists.

Write a config file early. Once your command has more than four flags, move them to config.yaml and run vllm serve --config config.yaml. It documents your choices, and a diff of the file shows exactly what changed.

Know where you would go next. If vLLM is more than you need on a laptop, Ollama and llama.cpp are lighter. If you want to compare servers, read Text Generation Inference and SGLang.

Practice cards

Card 1: Break it on purpose. Start Qwen3-0.6B with --max-model-len 512. Send a prompt of about 1,000 words. Read the 400 message and explain each number in it.

Card 2: Budget it. Start with utilization 0.5, note the KV cache size in the log, then 0.8, then 0.92. Write the three numbers in a table and describe the relationship in one sentence.

Card 3: Two servers, one GPU. Start one server at 0.45 on port 8000 and another at 0.45 on port 8001 with a different model. Confirm both answer. Then try 0.9 for the second and read the free-memory error.

Card 4: Rename the model. Serve with --served-model-name, call it with the old name and read the 404, then call it with the new one.

Card 5: Lock it. Add an API key from the environment, call without it, call with it, and then find one path that is not covered by it, such as the health route. Explain why a gateway is still needed.

Card 6: Template matters. Send the same question to /v1/completions as a raw prompt and to /v1/chat/completions as a message. Compare the quality.

Card 7: Container with volumes. Run the Docker command with both volumes, time the first and second starts, and write down the difference.

Card 8: Metrics. With a server running, curl /metrics, find the running and waiting request gauges, and fire ten concurrent requests to watch them move. Then read about scraping them in the Prometheus guide.

A short checklist

Before you start a server:

• I know my GPU, its total memory and the driver version.
• Nothing else is using the GPU (nvidia-smi is quiet).
• I am in a clean environment with vllm==0.30.0.
• I picked --max-model-len from my real needs.
• I set --host, and it is 127.0.0.1 unless a gateway exists.
• My key comes from VLLM_API_KEY, not from the command line.
• If the model is gated, I accepted the licence and exported HF_TOKEN.

After it starts:

• /health returns 200 and /v1/models shows the name I expect.
• I read the weights memory and KV cache size in the log.
• A chat request works with the served model name.
• I send temperature and max_tokens explicitly.

Before I call it done:

• Prompts go to the chat endpoint, or through llm.chat.
• Offline scripts have the main guard and batch their prompts.
• In Docker, I pinned the tag, mounted caches and set shared memory.
• Nothing sensitive is exposed beyond a private network.

When something fails, in order: read the line above the wrapper error, check nvidia-smi, check the flag against vllm serve --help=<flag>, rerun with VLLM_LOGGING_LEVEL=DEBUG, and only then search the web, adding your version number to the search. The neighbouring guides on Docker and Kubernetes are where to go when the container side is the part that is confusing.

When something still does not work

If you have followed the entries above and a start still fails, narrow the problem by changing one thing at a time. First, prove the GPU and driver are fine without vLLM: nvidia-smi should list your card and show memory. Second, prove the install is intact: python -c "import vllm; print(vllm.__version__)" should print 0.30.0. If the import fails, the problem is the environment, and rebuilding it is faster than diagnosing it. Third, prove the pipeline with the smallest model and the smallest context, for example vllm serve Qwen/Qwen3-0.6B --max-model-len 2048. If that works, grow one setting at a time until it breaks; the setting you just changed is the cause.

Keep a short notebook of what each failure looked like and what fixed it. Beginners who do this stop repeating mistakes within a week, because the same four or five messages account for most of what they will see. When you ask a colleague or the community for help, include the output of vllm collect-env, the exact command you ran, and the lines of the log from just before the error. A question with those three things attached usually gets answered in minutes, while "vLLM does not work" gets a request for exactly that information.

This is part two of three. It picks up where Beginner ended, with a model serving on one GPU and a client calling it, and takes you to the point where you can predict what vLLM will do, size it on purpose, and run it behind real traffic. The version throughout is vLLM 0.30.0, released on 22 September 2026. vLLM ships a minor release roughly every two weeks and removes things on minor releases, so every flag below was checked against that version, and the section on upgrades tells you how to keep it that way.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run on a real deployment.

What happens inside a vLLM instance

Beginner told you that vLLM batches requests continuously and stores attention state in blocks. To tune it you need the next layer of detail: which processes exist, who talks to whom, and what the scheduler decides on each iteration.

A running vllm serve is not one process. There are four kinds. The API server process handles HTTP, tokenizes prompts, loads images and audio, and detokenizes and streams the output. The engine core runs the scheduler, owns the KV cache bookkeeping and coordinates the GPUs, in a busy loop. One GPU worker per GPU loads weights and runs forward passes, and each worker contains a model runner that prepares inputs and captures CUDA graphs (Model Runner V2 has been the default for every model since 0.29). If you run data parallel replicas there is also a DP coordinator. The API server and the engine core talk over ZMQ; workers talk to each other over NCCL.

The counts follow a formula worth memorising because it explains surprising ps output: API servers equal the data-parallel size, engine cores equal the data-parallel size, workers equal tensor parallel times pipeline parallel times data parallel, plus one coordinator when data parallel is above one. Tensor parallel 4 gives six processes. Tensor parallel 2 with data parallel 4 gives seventeen.

Why does the split matter at work? Because the API server is CPU work, and the engine core is on the critical path of every token. When you serve many small requests, or heavy multimodal inputs, the API server can become the bottleneck while the GPUs sit idle. The flag --api-server-count adds API server processes for exactly that case. Reach for it when GPU utilization is low but latency is high and one CPU core is pinned, not before.

One scheduler step

The scheduler in the V1 engine does not distinguish a "prefill phase" from a "decode phase". On every iteration it has a token budget, max_num_batched_tokens, and a cap on concurrent sequences, max_num_seqs. It fills the budget in a fixed order. Running requests that need one more decode token go first, one token each. Then it admits waiting requests and gives them prompt tokens until the budget is spent. A prompt longer than the remaining budget is chunked: it gets what fits this step and continues on the next. Chunked prefill is on by default, so a long prompt never freezes everyone else's token stream.

This is the single most useful mental model for tuning. Prefill is compute-bound and decides time to first token (TTFT). Decode is memory-bandwidth-bound and decides inter-token latency (ITL). Both compete for the same budget on each step, and nearly every scheduling flag in this guide is a way of choosing who wins.

KV blocks and the block table

The KV cache is the stored attention keys and values for every token each live request has seen. vLLM does not give each request one contiguous slab. It divides GPU memory into fixed-size blocks (the token count per block is --block-size) drawn from a pool, and each request holds a block table mapping its logical token positions to physical blocks, much like an operating system's page table. Nothing is wasted on fragmentation, requests can grow one block at a time, and, importantly for later sections, two requests can point at the same physical block when they share a prefix.

When the pool is exhausted mid-generation, vLLM preempts: it evicts a request's blocks and, when memory frees, recomputes that request from its prompt. The V1 engine always recomputes rather than swapping to CPU, which is why the old --swap-space flag no longer exists. Preemption is not an error. It is a performance warning that your KV budget is too small for your concurrency, and you will learn to watch for it.

The memory budget as arithmetic

The most common production failure with vLLM is not a bug. It is a memory budget that does not add up. Beginner showed you two flags; here is the arithmetic behind them, so you can size a deployment before you pay for the GPU.

--gpu-memory-utilization is the fraction of the GPU's total memory that one vLLM instance may use for weights, activations and KV cache combined. The default in 0.30.0 is 0.92, not the 0.9 that older material quotes. It is a per-instance cap, not extra memory on top of the model, and it is a fraction of the whole device, not of what happens to be free. Everything left after weights and activation peaks becomes the KV cache.

KV budget  =  total GPU memory x utilization
              - weights - peak activations - CUDA graph and runtime overhead

That budget, not the weights, is what limits how many requests you can run at once. To convert it into tokens you need the per-token KV size, which depends on the model's architecture:

bytes per token = 2 (K and V) x layers x KV heads x head dim x bytes per element

Take a model with 32 layers, 8 KV heads and a head dimension of 128, which is the shape of Llama 3.1 8B (confirm yours in config.json). In BF16, two bytes per element, that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB per token. On an 80 GB card at 0.92 you have about 73.6 GB to share. The weights take roughly 15 GiB, activations and overhead take a few more, and what remains, perhaps 50 GiB, holds around 400,000 tokens. That is a sketch, not a promise, because overhead varies, but it tells you the order of magnitude: a few hundred thousand tokens of KV across all concurrent requests. Sixty-four concurrent requests of 8,000 tokens each fit comfortably. Sixty-four of 32,000 tokens do not, and the scheduler will preempt.

Note what the arithmetic says about --max-model-len. It does not reserve memory per request, so lowering it does not make each request cheaper. What it does is set the longest request vLLM must be able to hold, and at startup vLLM checks that one full-length request fits. If it does not, you get the most common startup error:

ValueError: To serve at least one request with the model's max seq len (131072),
(16.00 GiB KV cache is needed, which is larger than the available KV cache memory
(9.23 GiB). Based on the available memory, the estimated maximum model length is
75584. Try increasing `gpu_memory_utilization` ... or decreasing `max_model_len` ...

Read it as three numbers: what the model's default context would need, what you have, and the largest length that would fit. The estimate in the message is a usable --max-model-len. The fixes in order of preference are to set that length to what your application really needs, quantize the KV cache or the weights, add GPUs with tensor parallelism, and only then to raise the utilization.

Two sibling errors point elsewhere. No available memory for the cache blocks means weights plus activations already exceed the budget, so you need a bigger GPU, quantization or more parallelism. Free memory on device cuda:0 ... is less than desired GPU memory utilization (0.92 ...) means another process holds the GPU; run nvidia-smi first, and if you want two instances on one card, give each a share that sums below one, such as 0.45 each.

Scheduling knobs: concurrency, token budget and load shedding

Once the memory budget is sound, the next question is how the scheduler spends it. Three settings do most of the work: --max-num-seqs, --max-num-batched-tokens and the choice about chunked prefill.

--max-num-seqs caps how many sequences run in one step. --max-num-batched-tokens caps how many tokens, prompt and output together, the engine processes in one step. The defaults are not constants: in 0.30.0 they depend on the GPU and on whether you are using the server or the offline LLM class. On a 160 GB-class card such as a B200, both entrypoints get 1,024 sequences and a 16,384-token budget. On an H100 or H200 (70 GB or more, not A100), you get 1,024 sequences and 16,384 tokens offline but 8,192 tokens in the server. On everything else, including A100, L4 and consumer cards, it is 256 sequences, with 8,192 tokens offline and 2,048 tokens in the server. If you benchmark with the offline class and deploy with the server on a smaller GPU, you have quietly changed the token budget by a factor of four.

What does the token budget do? A small budget, around 2,048, means each step is short, so decodes are never held up behind a large prefill and ITL is smooth. A large budget means prompts are absorbed faster, which lowers TTFT and raises throughput, and the vLLM tuning guidance suggests values above 8,192 for small models on large GPUs. The trade-off is that a step full of prefill delays every decode in that step. You tune it by deciding which latency your users feel. A chat product with streaming cares about ITL and a snappy first token; a batch summarisation job cares about total throughput and nothing else.

Chunked prefill is on by default, and the practical advice is to leave it on. If you disable it, max_num_batched_tokens must be at least max_model_len, because a whole prompt must fit in one step, and long prompts then stall everything behind them. For the rare case where one enormous prompt arrives among short ones, --long-prefill-token-threshold lets you treat prompts above a length as long and limit how much of each step they can consume.

Preemption is the symptom, not the disease

When concurrency times average sequence length exceeds the KV capacity, requests are preempted and recomputed. The log carries a warning that a request was preempted by recompute because there is not enough KV cache space (the exact wording has varied between releases), and the metric vllm:num_preemptions counts them; 0.30 adds a vllm:request_num_preemptions histogram so you can see how many times individual requests were hit. A handful is fine. A steady stream means p99 latency is quietly inflated, because each preempted request redoes its prefill. The remedies come from the memory arithmetic: more KV (higher utilization, FP8 KV cache, quantized weights, more GPUs) or less demand (lower --max-num-seqs, shorter --max-model-len, a smaller token budget).

Shed load, do not queue forever

By default waiting requests queue without limit, which turns a traffic spike into a latency spike that outlasts the spike itself. Since 0.29 you can bound the queue with --max-num-queued-reqs and --max-num-queued-tokens. Requests beyond the limit are rejected quickly, which lets a gateway retry elsewhere or return an honest error. A fast "busy" is almost always better for users and for your autoscaler than a ninety-second wait. Priority is also available: --scheduling-policy priority lets requests carry a priority, so an interactive request can overtake a bulk job in the queue.

Configuration you can review and reproduce

By now your launch command has a dozen flags and lives in someone's shell history. That is how "it worked on my machine" begins. vLLM has over two hundred and fifty flags, and the way to keep them manageable is a file in version control.

vllm serve --config config.yaml reads a YAML file whose keys are the long flag names without the dashes prefix:

model: meta-llama/Llama-3.1-8B-Instruct
host: "127.0.0.1"
port: 8000
max-model-len: 16384
gpu-memory-utilization: 0.9
served-model-name: llama-8b

Precedence is command line over config file over defaults. A positional model on the command line wins over model: in the file. This is deliberate and useful: the file is the reviewed baseline for an environment, and an operator can override one value for an experiment without editing it. It also means the file can be misleading if someone appends a flag in a Kubernetes manifest, so print the effective configuration from the startup log when you are debugging.

Some flags take JSON, such as --compilation-config, --speculative-config and --kv-transfer-config. You can pass a JSON string, or use dot notation to set one field: --compilation-config.cudagraph_capture_sizes+ 1 appends to a list. Quote JSON carefully in shell and YAML; most "invalid argument" reports at this level are quoting. Numbers accept human-readable forms, so --max-model-len 32k means 32,000 while 32K means 32,768, a distinction that matters. The value auto or -1 asks for automatic selection where a flag supports it.

Generation defaults come from the model

The change that surprises teams most is not a flag but a behaviour. Since v0.8.0 the server takes default sampling parameters from the model's generation_config.json. If a model repository ships temperature: 0.6 or a top_p, requests that do not set those values inherit them. When you upgrade, or swap to a fine-tune with a different file, outputs change although your code did not. The default setting is --generation-config auto. To get vLLM's own defaults, pass --generation-config vllm, and to force specific values pass --override-generation-config '{"temperature":0.6}'. In production, set sampling parameters explicitly in your requests and make the server's defaults a deliberate choice, not an accident of the checkpoint.

Automatic prefix caching

Real workloads repeat themselves. A system prompt is identical across every request, a retrieval-augmented chat repeats the same documents, and a multi-turn conversation resends its whole history each time. Recomputing the KV for those repeated prefixes is pure waste, and prefix caching removes it.

Automatic prefix caching (APC) is on by default in 0.30.0. Because KV is stored in blocks, a block can be identified by a hash of its contents: the hash of the parent block, the tokens in this block, and any extra identity such as a LoRA adapter ID, multimodal hashes, or a cache salt. When a new request arrives, vLLM walks its prompt block by block and, for every block whose hash is already in the pool, points the request at the existing physical block instead of computing it. Only the first block that misses, and everything after it, is computed.

Two properties follow directly. First, matching is by prefix: a one-token change early in the prompt changes every later block's hash, so the cache hit ends there. This is why you put the stable part of the prompt first (system prompt, tool definitions, documents) and the variable part last (the user's question). Interleaving a timestamp or a request ID near the top of a prompt silently disables the cache for the whole request. Second, the cache accelerates prefill only. It lowers TTFT and reduces compute, but decode speed is unchanged.

Cached blocks that no running request is using stay in memory until the space is needed, and are evicted least-recently-used first. That is why a long-lived server with spare KV gets better hit rates than a freshly restarted one, and why a restart in the middle of the day costs you more than the startup time. Hit rate is observable: vllm:prefix_cache_hits divided by vllm:prefix_cache_queries is the ratio, and it should be a graph on your dashboard. The hash algorithm is selectable with --prefix-caching-hash-algo (default sha256; sha256_cbor, xxhash and xxhash_cbor are the alternatives). The default is the collision-resistant choice, which matters for the next point.

Caching and multi-tenancy

A shared prefix cache is a shared resource. If tenant A can detect that a prompt prefix is already cached, by timing the first token, they learn something about what tenant B sent. When tenants must be isolated from each other, pass a per-tenant cache_salt in the request. The salt is mixed into the block hash, so tenants with different salts never share blocks, at the cost of losing cross-tenant hits for common prefixes. For a single-company deployment serving internal teams, sharing is usually fine and the hit rate is worth more; for a platform serving external customers, use salts.

Quantization and the KV cache dtype

Quantization trades numeric precision for memory and sometimes speed. At this level the question is never "should I quantize" in the abstract; it is which of three different things you are shrinking, because they fix different problems.

Weights are the model parameters. Quantizing them (FP8, AWQ, GPTQ, GGUF, NVFP4, compressed-tensors checkpoints made with llm-compressor, and others) shrinks the fixed cost, lets a larger model fit on a given GPU and leaves more room for KV. The KV cache is the per-request working memory. Setting --kv-cache-dtype fp8 roughly halves the bytes per token compared with BF16, which roughly doubles how many tokens of context you can hold, with a small accuracy risk on long contexts. Activations are usually left alone.

For checkpoints that were quantized offline, you usually do not need a flag: vLLM reads the quantization method from the checkpoint config. For a BF16 model you can ask for online FP8 with --quantization fp8. Two changes in recent releases bite real deployments. Support for GPTQ checkpoints that use activation ordering (desc_act, the g_idx path) was removed in 0.30, so those checkpoints no longer load and you should requantize or switch to AWQ or FP8. And bitsandbytes became an out-of-tree plugin in 0.28, so it must be installed separately.

The practical order of operations is the cheapest, safest change first. Try the FP8 KV cache when your pain is context length or concurrency, since it does not touch the weights. Try FP8 weights on hardware with native FP8 support when you need a model to fit or want more throughput. Keep a quality evaluation, even a small one, that you rerun on each change, because "the benchmark says fine" and "our users' prompts are fine" are different claims. Remember also that on a T4 or other compute-capability-7.5 card BF16 is not supported; you get Bfloat16 is only supported on GPUs with compute capability of at least 8.0, and the fix is --dtype half.

Tool calling, reasoning models and structured outputs

Applications rarely want free text. They want a function call to execute, a JSON object to parse, or the final answer separated from the model's chain of thought. vLLM does these things in the server, and each one has a precise configuration that fails in specific ways.

Tool calling

For the OpenAI tools parameter to work, vLLM needs to know how your model writes tool calls, because every model family uses a different text format. You tell it with two flags together:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-auto-tool-choice --tool-call-parser llama3_json \
  --chat-template examples/tool_chat_template_llama3.1_json.jinja

--tool-call-parser selects the parser. The available names include hermes, llama3_json, llama4_pythonic, mistral, deepseek_v3, deepseek_v31, qwen3_xml, openai (for gpt-oss), pythonic, granite and phi4_mini_json, with more than twenty in all, and --tool-parser-plugin registers your own. The parser must match the model family; picking hermes for a Llama checkpoint produces text that never becomes a tool_calls entry. The chat template matters too, because the template is what renders the tool definitions into the prompt. Use the template the model card or the vLLM examples recommend, not a guess.

Two errors are worth recognising on sight. If you send tool_choice: "auto" to a server started without the flags, you get "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set. If you start the server with only --enable-auto-tool-choice, it refuses to start with --enable-auto-tool-choice requires --tool-call-parser. The tool_choice values supported are auto, required, none, and a named function.

Reasoning models

Models that think before answering emit their reasoning in the same text stream as the answer. --reasoning-parser <name> (for example deepseek_r1 or qwen3) splits them, and the reasoning arrives in a reasoning_content field of the response, separate from content. The parser name must match the model family. Without it your users see the thinking text, and your JSON parser sees it too.

Structured outputs

When you need valid JSON, a regular expression match or a choice from a list, constrained decoding guarantees the format by masking tokens the grammar does not allow. The syntax changed in v0.12.0 and old examples are still everywhere: the request fields guided_json, guided_regex, guided_choice and guided_grammar are replaced by a single structured_outputs object, and GuidedDecodingParams became StructuredOutputsParams. The current form, through the OpenAI client:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "confidence": {"type": "number"},
    },
    "required": ["sentiment", "confidence"],
}

resp = client.chat.completions.create(
    model="llama-8b",
    messages=[{"role": "user", "content": "Classify: the delivery was late again."}],
    extra_body={"structured_outputs": {"json": schema}},
)
print(resp.choices[0].message.content)

The vLLM-specific field goes in extra_body, because the OpenAI SDK does not know about it. The same constraint also works through the OpenAI-native response_format={"type": "json_schema", ...}. The other modes are choice, regex, grammar and structural_tag. Offline, you pass SamplingParams(structured_outputs=StructuredOutputsParams(json=schema)), importing the class from vllm.sampling_params. The grammar engine is selected with --structured-outputs-config.backend, whose options are xgrammar, guidance, outlines, lm-format-enforcer and auto (the default); the old --guided-decoding-backend flag is gone.

Constrained decoding guarantees syntax, not correctness. The model can still put the wrong sentiment in a perfectly valid object. Constraints also add some per-request overhead, so test with your real schemas and measure.

Serving many fine-tunes with LoRA

A team that fine-tunes per customer or per task faces an expensive default: one GPU deployment per fine-tune. LoRA (low-rank adaptation) fine-tunes produce small adapter files that modify a shared base model, and vLLM can serve many adapters on one base model at the same time, mixing requests for different adapters into the same batch.

vllm serve meta-llama/Llama-3.2-3B-Instruct --enable-lora \
  --lora-modules sql-lora=jeeejeee/llama32-3b-text2sql-spider \
  --max-loras 4 --max-lora-rank 64

--lora-modules registers adapters by name; a JSON form also accepts name, path and base_model_name. A client selects an adapter by sending its name as the model field, and the base model remains available under its own name. Two limits trip people. --max-loras is how many distinct adapters can be active in one batch, default 1, and --max-lora-rank is the largest adapter rank the server will accept, default 16. An adapter trained at a higher rank than the server's limit will not load. Set the rank limit to the largest you will use, but no higher, since it sizes memory for the adapter slots. --max-cpu-loras controls how many adapters are kept warm in CPU memory.

The economic argument is straightforward. If ten customers each need a fine-tune of the same 8B base model, ten dedicated GPUs run mostly idle, while one server with --max-loras set appropriately shares the base weights and the KV cache. The cost is the operational care of managing adapters and some per-request overhead. Prefix caching remains correct, because the adapter ID is part of the block hash, so adapters never see each other's cached blocks.

Containers and Kubernetes

A laptop command becomes a product the moment it is packaged. The official image vllm/vllm-openai runs vllm serve for you, and everything after the image name is passed through as server arguments. If Docker itself is new ground, the Docker guide covers the mechanics; this section covers what is specific to vLLM.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:v0.30.0 \
  --model Qwen/Qwen3-0.6B

Each line earns its place. The Hugging Face cache mount means weights download once, not on every container start. The second mount persists the torch.compile cache at /root/.cache/vllm; without it every container recompiles the model, which is a large part of the startup you will be paying for. --ipc=host (or a large --shm-size) gives PyTorch and NCCL the shared memory they need, and it is needed whenever you use more than one GPU (--tensor-parallel-size). And the tag is pinned. latest is for demos: it moves every two weeks, and the changes between releases include removed flags and changed defaults.

Kubernetes

On Kubernetes a vLLM replica is a stateful-feeling Deployment: slow to start, expensive, holding a warm cache. The Kubernetes guide explains the objects; here are the vLLM-specific parts of a Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-qwen
spec:
  replicas: 1
  selector:
    matchLabels: { app: llm-qwen }
  template:
    metadata:
      labels: { app: llm-qwen }
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:v0.30.0
          args: ["--config", "/config/config.yaml"]
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef: { name: hf-token, key: token }
            - name: VLLM_API_KEY
              valueFrom:
                secretKeyRef: { name: vllm-api-key, key: key }
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
          volumeMounts:
            - { name: shm, mountPath: /dev/shm }
            - { name: config, mountPath: /config }
            - { name: model-cache, mountPath: /root/.cache/huggingface }
            - { name: vllm-cache, mountPath: /root/.cache/vllm }
      volumes:
        - name: shm
          emptyDir: { medium: Memory }
        - name: config
          configMap: { name: llm-qwen-config }
        - name: model-cache
          persistentVolumeClaim: { claimName: model-cache }
        - name: vllm-cache
          emptyDir: {}

Several decisions in that manifest come from real failures. The startup probe has a generous failure threshold (here fifteen minutes) because /health only answers once the engine is ready, and a liveness probe without it will kill the pod mid-load, leaving only KeyboardInterrupt: terminated in the logs. The in-memory /dev/shm stands in for --ipc=host and prevents NCCL error: unhandled system error, the failure a container shows when shared memory is missing. The secrets come from secretKeyRef, not from arguments, so they do not appear in process listings or manifests; prefer the VLLM_API_KEY environment variable to --api-key for that reason. The model cache lives on a volume so a rescheduled pod does not re-download tens of gigabytes. For GCC regions this matters more than it sounds: in the Gulf and in Egypt, egress and download speed from the Hub vary by provider, and some employers also require weights to stay in a regional bucket. Copy approved weights into storage in your region and point --model at that path.

Scaling replicas is where vLLM differs from a stateless web service. A new replica starts cold, so adding capacity takes minutes and begins with an empty prefix cache. Round-robin routing scatters requests that share a prefix across replicas and wastes the cache. The vLLM production-stack project provides a router with session, prefix-aware and KV-aware routing modes for exactly this, and KServe, llm-d and others integrate vLLM as a runtime. If your organisation already uses KServe, see the KServe guide for how it wraps a runtime like this one. Do not autoscale on CPU: a GPU server's CPU is nearly flat. Scale on queue depth, covered in the observability section.

Securing an endpoint properly

--api-key feels like security, and the official security page is explicit that you should not rely on it alone. It protects the OpenAI-style API prefixes (/v1, /v2, /inference and the Cohere prefix). It does not cover every route: /invocations, /tokenize and a handful of operational endpoints are outside it. So treat the key as a convenience between friendly clients, and put a real gateway in front.

The pattern is a reverse proxy or API gateway (NGINX, Envoy, a cloud gateway, or LiteLLM, which also gives you per-key budgets; see the LiteLLM guide) that authenticates each caller and allowlists the exact paths you expose, typically /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models. Everything else, including /metrics and /tokenize, is denied at the edge. Health checks come from the orchestrator inside the network, not from the internet. vLLM itself has no per-user accounting or rate limits, so the gateway is also where quotas, request size limits and a cap on max_tokens live.

Next, the surfaces that should never exist in production:

• VLLM_SERVER_DEV_MODE=1 exposes endpoints such as /collective_rpc, which is arbitrary remote procedure calls into your workers.
• --enable-tokenizer-info-endpoint and --profiler-config are development tools.
• Runtime LoRA updating is a file-loading endpoint; enable it only if the network path to it is private.
• --trust-remote-code executes Python from the model repository. Use it only for repositories you have read, and pin --revision to a commit so the code cannot change under you.
• --trust-request-chat-template lets clients supply their own chat template; it is off by default and should stay off for untrusted clients.

Traffic between nodes in a multi-GPU or multi-node deployment is unauthenticated and unencrypted by default, and the gRPC server enabled by --grpc is unauthenticated too. These belong on a private network segment, never on a public interface. Set --host explicitly: the default binds to all interfaces, and a development command copied into a VM with a public IP is a classic way to publish a free GPU. CORS defaults to --allowed-origins ['*']; tighten it if browsers call the server directly. TLS is available through --ssl-keyfile and --ssl-certfile, though terminating at the proxy is more common.

Metrics, logs and autoscaling signals

An LLM server can be healthy by every ordinary measure and still be failing users: the process is up, CPU is low, and requests wait ninety seconds. The signals that matter are in the engine, and vLLM exposes them as Prometheus metrics at /metrics. That endpoint is unauthenticated, which is the reason it must stay behind your network boundary and be scraped from inside the cluster. The Prometheus guide and Grafana guide cover the tooling.

Group the series by the question they answer:

Reading them together is the skill. Waiting requests rising with KV usage low points at the sequence cap or token budget. Waiting requests rising with KV usage near full and preemptions climbing means memory is the limit. High TTFT with a healthy queue means slow prefill, from long prompts or no cache hits. High ITL with healthy TTFT means large prefills are crowding decode steps, so shrink the token budget.

For autoscaling, use vllm:num_requests_waiting or KV cache usage, not CPU or GPU utilization percentage, which stays high whenever any work exists. Scale-up takes minutes because of startup, so set the trigger with headroom, and scale down slowly. Deprecated metrics are hidden one minor release after deprecation; if a dashboard goes blank after an upgrade, --show-hidden-metrics-for-version=X.Y brings them back temporarily while you migrate.

Benchmarking and testing before you promote

Every tuning decision in this guide is a hypothesis until you measure it. vLLM ships its own benchmark tool for exactly that, so there is no excuse for guessing. The subcommands are vllm bench latency, throughput, serve and sweep. For a deployed server, the one that matters is serve, because it sends concurrent requests to a running endpoint the way real clients do:

vllm bench serve --model Qwen/Qwen3-0.6B \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --num-prompts 500

It reports throughput and the latency distributions you care about: TTFT, ITL (or TPOT) and end-to-end latency, with percentiles. Read the p95 and p99, not the mean, since the tail is where preemption and queueing show up. A random dataset is fine for comparing configurations against each other, but it hides the effect of prefix caching and of your real length distribution. For a result you can quote to a product team, replay a sample of real prompts from logs (with consent and with sensitive data removed) so the input and output lengths and the shared prefixes look like production.

Testing the service in CI

You usually cannot run a 70B model in a pipeline, so test in layers. Use a tiny model such as Qwen/Qwen3-0.6B or facebook/opt-125m, on CPU or a small GPU, to check what does not depend on model quality: that the server starts with your config.yaml, that /health returns 200, that the OpenAI client works against /v1/chat/completions, that your tool-call and structured-output request shapes are accepted, and that the gateway's allowlist rejects what it should. A smoke test like this catches removed flags on the day you bump the version, which is when it is cheapest to fix.

import subprocess, time, urllib.request

def wait_ready(url, timeout=600):
    end = time.time() + timeout
    while time.time() < end:
        try:
            if urllib.request.urlopen(url).status == 200:
                return
        except Exception:
            time.sleep(5)
    raise TimeoutError("server never became ready")

def test_server_starts_and_answers():
    proc = subprocess.Popen(["vllm", "serve", "Qwen/Qwen3-0.6B", "--max-model-len", "2048"])
    try:
        wait_ready("http://localhost:8000/health")
        body = urllib.request.urlopen("http://localhost:8000/v1/models").read()
        assert b"Qwen/Qwen3-0.6B" in body
    finally:
        proc.terminate()
        proc.wait(timeout=60)

Debugging beyond the basics

Beginner taught you to read the error. At this level the failures are subtler: a server that starts but is slow, hangs, or misbehaves under load. The method is the same as with any system, which is to localise first and tune second.

The startup wrapper error. RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {...} is not the cause; it is the engine reporting that a child process died. Scroll up, usually one to three exceptions above the wrapper, to find the real error: a download or authentication failure, an unsupported model architecture, or one of the memory errors from earlier.

Isolate the engine from the GPU kernels. If you suspect a CUDA graph problem, such as a crash near self.graph.replay(), restart with --enforce-eager. If the crash disappears, you have localised it and have a workable mitigation while you file a report. Run vllm collect-env and paste its output into any bug report, since version mismatches are the most common root cause behind "weird" failures.

Broken binaries after pip install. vLLM wheels are built against one specific PyTorch, in 0.30.0 torch 2.13.0. If something later installs a different torch into the same environment, importing vllm._C fails with undefined symbol errors. Do not patch around it. Make a clean virtual environment and reinstall vLLM with uv pip install vllm --torch-backend=auto, letting it bring its pinned torch.

Upgrades without surprises

vLLM follows semantic versioning but removes things on minor releases, and minor releases arrive about every two weeks. The deprecation policy is a three-step pipeline: a feature is first deprecated and still on by default, with a warning that names the removal version; then deprecated and off by default, with an opt-in flag; then removed. It applies to flags, environment variables, config files, server APIs and public Python APIs. The practical lesson is that the deprecation warnings in your logs are a to-do list with deadlines, so grep for them in CI.

The workflow that survives all of this:

1. Pin both the Python package (vllm==0.30.0) and the container tag.
2. Read the release notes' breaking-changes section before bumping, and search your manifests for each flag and variable they mention.
3. Upgrade in a clean environment, because vLLM pins its PyTorch.
4. Run the smoke test, then the evaluation set, then vllm bench serve against the previous version's numbers.
5. Roll out as a canary or blue-green. A restart discards the in-flight requests and the prefix cache, so a rolling update shows up in latency graphs.

Putting it all together

Here is how the pieces combine into one coherent deployment: an internal assistant that answers from company documents with tool calls, serving a mid-size model to about a hundred employees in the Gulf, on one 8-GPU node, with company data staying in-region.

The deployment starts from a reviewed config.yaml kept in version control, with the checkpoint pinned by --revision and the weights copied to regional storage. The server runs the pinned vllm/vllm-openai:v0.30.0 image with a startupProbe, in-memory /dev/shm, persistent compile and model caches, and secrets from the secret manager. Tensor parallel size (--tensor-parallel-size) is the smallest that fits the model with room for KV; if one replica uses four GPUs, two replicas fill the node, behind a gateway that authenticates employees, allowlists four paths and caps max_tokens.

Prompts are shaped for the cache: system prompt and tool definitions first, retrieved passages next, the user's question last, so the prefix cache turns the repeated parts into near-free prefill. Sampling parameters are set explicitly, with low temperature for tool calls, and structured outputs constrain the arguments. A bounded queue turns overload into fast, retryable errors. Prometheus scrapes /metrics from inside the cluster, with panels for queue, KV usage, preemptions, TTFT, ITL and prefix hit rate, and alerts on sustained waiting requests and any steady preemption.

Before every upgrade, the CI pipeline starts a tiny model with the production config.yaml, runs the smoke test, replays the regression prompts, and compares vllm bench serve results with the last release. Capacity numbers (KV tokens, concurrency at the latency target) sit in the runbook for the next person.

Where you are now

You can predict what vLLM does rather than discovering it. You know the process layout, you can turn a GPU's memory into a token budget on paper, and you can read the scheduler's behaviour from three flags and a handful of metrics. You configure with a reviewed file, you cache prefixes on purpose, you pick quantization for the right resource, and you serve tools, reasoning models, structured outputs and many LoRA adapters from one deployment. You deploy with pinned images, correct probes and shared memory, protect the endpoint at a gateway, watch the queue and the KV cache, and upgrade with a smoke test and an evaluation set in front of the change.

What the next level adds is the platform view: tensor and pipeline parallelism across nodes, expert parallelism, speculative decoding, disaggregated prefill, cache-aware routing and incident playbooks. Compare vLLM with TGI and SGLang once you can describe your own bottleneck precisely.

Sources

• vLLM documentation
• OpenAI-compatible server
• Architecture overview
• Conserving memory
• Automatic prefix caching
• Tool calling
• Structured outputs
• LoRA adapters
• Parallelism and scaling
• Security
• Production metrics
• vLLM 0.30.0 release notes

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic, for an interviewer who knows you have run vLLM in production. At this level nobody asks what PagedAttention is called. They give you a symptom and watch how you reason. Senior reviews all three. The version here is vLLM 0.30.0.

Where this picks up

Foundations, in one screen

Say it in this order. Each step the scheduler fills a token budget (max_num_batched_tokens) and a sequence cap (max_num_seqs): running decodes first, then waiting prompts, with long prompts chunked. KV cache lives in fixed-size blocks from a pool, with a block table per request, so memory is not fragmented and shared prefixes can share blocks. Concurrency is limited by KV capacity, not by weights. When the pool runs out, requests are preempted and recomputed. Prefill drives TTFT and is compute-bound; decode drives ITL and is memory-bandwidth-bound.

Scenario questions

"The service is slow at peak. What do you check first?"

A strong answer refuses to guess and goes to the engine metrics, in an order that splits the problem in half. First, is there a queue? Look at vllm:num_requests_waiting and vllm:request_queue_time_seconds. If the queue is empty, the slowness is inside a request, so compare prefill time and decode time. Long prefill with a cold cache points at prompt shape or lost prefix hits. Slow decode with good TTFT points at crowded steps.

If there is a queue, ask whether memory is the limit by checking vllm:kv_cache_usage_perc and vllm:num_preemptions. Near-full KV with rising preemptions means the instance is out of memory for its concurrency: more KV (FP8 KV cache, quantized weights, a bigger GPU, more tensor parallelism), or less demand (lower --max-num-seqs, shorter --max-model-len). Low KV usage with a queue means a scheduling cap is the limit, and --max-num-seqs or the token budget is too small for the GPU.

What a weak answer does: "add more GPUs" or "raise gpu-memory-utilization" without evidence, or checks CPU and GPU utilization, which stay high whenever there is any work.

"Startup fails with a KV cache error. Walk me through it."

The message says the model's maximum sequence length needs more KV than is available and gives an estimated maximum model length. A strong answer reads three numbers: what the default context needs, what exists, and the length that fits. The best first fix is setting --max-model-len to what the application needs, because most services never use the model's full 128k context. Then FP8 KV cache, quantized weights, tensor parallelism, and only after those a higher utilization, which has a ceiling and leaves little headroom. Mention that the estimate in the message is directly usable.

What a weak answer does: treats it as a vLLM bug, or keeps raising the utilization until the card has no headroom left.

"We upgraded and output quality changed. Nothing in our code changed."

Four suspects, in order of likelihood. First, generation defaults: since v0.8.0 the server takes sampling defaults from the model's generation_config.json, so a new checkpoint or revision can change temperature and top-p. Check --generation-config, and set sampling parameters in the request. Second, the chat template changed with the tokenizer. Third, a quantization or kernel path changed between releases. Fourth, a default context length dropped (0.30 aligned YaRN with Transformers, which lowered some models' defaults) and long prompts now fail or truncate. The fix for all four is a regression set run before promotion, and pinned revision, image tag and explicit --max-model-len.

"The pod keeps restarting while loading the model."

Check the container log for KeyboardInterrupt: terminated. That is the signature of a probe killing the process before weights finished loading. The fix is a startupProbe on /health with a failure threshold large enough for the slowest cold start, with liveness and readiness only taking over after it. Then ask why startup is slow: model download each time (use a persistent Hugging Face cache or regional storage), compile each time (persist /root/.cache/vllm), network file system loading (copy to local disk).

"Two replicas, but throughput is no better than one."

Strong answers think about routing and the cache. Round-robin scatters requests that share a prefix, so each replica sees a cold cache and recomputes. Cache-aware or session-affine routing fixes that. They also check whether the bottleneck was the API server (CPU-bound tokenization, fixed with --api-server-count) or the clients themselves, because a load generator that cannot saturate two replicas will say nothing about them.

"A tool-calling request returns plain text instead of tool_calls."

Check the three pieces in order: --enable-auto-tool-choice is set, --tool-call-parser matches the model family, and the chat template renders tools for that model. A wrong parser name is the usual culprit; a template that ignores tools is next. If both are right, the model may be too small to produce reliable calls; validate arguments and consider constraining them with structured outputs. Know the two exact errors: using tool_choice: "auto" without the flags returns an error that both flags are required, and starting with only --enable-auto-tool-choice fails at startup.

"Is it safe to expose this endpoint to the internet?"

No, and the reason is specific. --api-key covers only the /v1, /v2 and /inference prefixes, and the official security page says not to rely on it alone. Other routes such as /tokenize and /invocations fall outside it, /metrics is unauthenticated, and vLLM has no per-user quotas. Put a gateway in front that authenticates, allowlists the few paths you expose, caps max_tokens and rate-limits. Mention that --host defaults to all interfaces, that dev mode and runtime LoRA updating must be off, and that inter-node traffic belongs on a private network.

Trade-off questions

Throughput or latency?

The token budget is the lever. A small budget (around 2,048) gives smooth ITL because decodes are never stuck behind a big prefill. A large budget (above 8,192 on a big GPU) raises throughput and lowers TTFT for bulk work. --performance-mode offers interactivity and throughput presets. A strong answer chooses by workload (a streaming chat product versus an overnight batch job), says to measure with vllm bench serve and defines success as goodput: throughput while still meeting the latency target.

Quantize weights, quantize the KV cache, or buy a bigger GPU?

They solve different problems. Weight quantization (FP8, AWQ and others) fixes "the model does not fit" and frees KV. FP8 KV cache (--kv-cache-dtype fp8) roughly doubles token capacity and does not touch the weights, so it is the cheapest first step for context length or concurrency problems. A bigger GPU or more tensor parallelism costs money but not quality. The senior part of the answer is the evaluation: whichever you pick, run your own prompts, not just a public benchmark. Add the 0.30 caveat that GPTQ checkpoints using activation ordering no longer load.

One server per fine-tune, or LoRA multiplexing?

LoRA shares the base weights and the KV cache across adapters, so ten low-traffic fine-tunes cost one GPU instead of ten. The costs are per-request overhead, the --max-loras and --max-lora-rank limits (defaults 1 and 16), and operational discipline: adapters must be named, versioned and loaded safely, and runtime loading needs VLLM_ALLOW_RUNTIME_LORA_UPDATING=True, which should not be reachable publicly. A dedicated server still wins when one tenant has heavy, steady traffic, needs isolation guarantees, or uses full fine-tuned weights instead of adapters.

Tensor parallelism or replicas?

Tensor parallelism splits one model across GPUs: it lets a large model fit and can lower latency, at the cost of communication every layer, which wants NVLink. Replicas multiply throughput with no per-step communication, and fail independently. The usual answer: the smallest tensor parallel size that fits the model with room for KV, then replicas for scale, with cache-aware routing in front.

Shared prefix cache or isolated tenants?

Sharing gives higher hit rates and lower cost; isolation prevents timing side channels where one tenant infers another's prompts from first-token latency. Per-tenant cache_salt is mixed into the block hash and isolates at the cost of cross-tenant hits. Internal single-company use can share; a platform serving separate customers should salt.

Pin everything, or follow latest?

Pin. vLLM releases about every two weeks and removes flags on minor releases. Pin the package or image tag and the model --revision, read breaking changes, upgrade in a clean environment, and run smoke, evaluation and benchmark gates, with a canary rollout. Following latest trades control for freshness you did not ask for. The counterweight a good answer admits: falling too far behind makes the eventual jump larger, so schedule upgrades.

What a strong answer covers that a weak one misses

Fast review: the things to have ready

Common interview questions

What does --gpu-memory-utilization actually limit? The fraction of the GPU's total memory that one instance may use for weights, activations and KV cache together. Default 0.92. Two instances on one card need shares that sum below one.

Why does lowering --max-model-len not shrink per-request memory? It does not reserve memory per request. It sets the longest request the instance must be able to hold, and vLLM checks that one such request fits in KV at startup. It frees nothing by itself, though a shorter limit lets startup succeed.

What is the difference between preemption and an out-of-memory error? Preemption is the scheduler evicting a request's KV blocks and later recomputing it, in V1 always by recompute. It costs latency but not correctness. An out-of-memory error at startup means the budget could not hold even one request.

Why does the order of the prompt matter? Prefix caching matches block by block from the start, so a stable prefix gets reused and a change early in the prompt ends the match. Put system prompts, tools and documents first and the user's input last.

How do you pass vLLM-specific parameters from the OpenAI SDK? In extra_body, for example extra_body={"structured_outputs": {"json": schema}} or {"top_k": 20}.

Why is the generation_config.json behaviour a risk? Because sampling defaults come from the checkpoint, and an upgrade or a different revision can change them without a code change. Set parameters explicitly, or use --generation-config vllm.

Why not name a Kubernetes Service vllm? Kubernetes injects VLLM_* variables for it, such as VLLM_PORT=tcp://..., which collide with vLLM's own internal variables.

What would you alert on? Sustained waiting requests, a steady preemption rate, KV usage near full, p99 TTFT and ITL against targets, and a falling prefix cache hit ratio.

How do you test a new version before promotion? A smoke test with a tiny model and the production config, a replayed regression set scored against the old version, and vllm bench serve compared with the previous release's numbers.

Why can the same model start on an H100 but fail on an A100 with the same flags? Memory and defaults both differ. The 80 GB cards may have equal capacity, but the hardware-dependent defaults for max_num_seqs and the token budget are not the same, and an A100 is not in the group that gets the larger values. Read the startup log of each rather than assuming the flags mean the same thing.

A client gets a 404 saying the model does not exist. What happened? The model field must match a name from /v1/models, which is the --served-model-name if you set one, or a LoRA adapter name. A client migrated from a hosted API still sending gpt-4 is the classic cause.

A request returns a 400 about maximum context length. Is the server broken? No. Prompt tokens plus requested output tokens exceeded max_model_len. Shorten the prompt, lower max_tokens, use truncate_prompt_tokens, or serve with a larger length if memory allows. After an upgrade, check whether the default length itself dropped.

Why does a restart in the middle of the day hurt more than its startup time? The prefix cache and every in-flight request are lost, and the replica comes back cold. That is why upgrades roll out as canaries with traffic shifted gradually.

When would you not use LoRA multiplexing? One tenant with heavy steady load, strict isolation needs, or fully fine-tuned weights.

Sixty-second self-test

Answer aloud, without notes.

1. Name the three process types in one vLLM instance and what each does.
2. Give the formula for KV bytes per token, and say why --max-model-len does not reserve per-request memory.
3. Which flag sets the token budget, what does a small value favour, and why do server and offline defaults differ?
4. What breaks a prefix cache hit, and what does cache_salt do?
5. What does a preemption mean, and which two metrics tell you it is happening?
6. Name the two tool-calling flags and one error you get when they are missing.
7. Give two reasons --api-key is not a security boundary.
8. Name three signals you would scale on, and one you would not.
9. List four things in a safe upgrade process.
10. Which removed settings do old blog posts still recommend?

Part two of three. At this level the model serves and the client gets answers, so the problems change character. They are no longer "why won't it start" but "why does p99 spike at lunchtime", "why did quality change after the upgrade" and "why is the second replica no faster". Start with the error table, then the practices in order of leverage, then practice cards and operational recipes. The version is vLLM 0.30.0.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In order of leverage: the first few change what you can see and what you can safely change, the later ones refine.

Why this order

Observability comes first because every other change needs a before and after. Pinning comes second because it makes every later experiment reproducible. Sizing KV on paper and shaping prompts for the cache are next because they change cost the most for the least risk: they need no new hardware and no new code. Gateways and queue limits protect you from the failure modes that hurt customers; testing keeps your changes honest. The last item, persisted caches, saves minutes per restart, which is little per incident and large over a year.

A word on what is not on the list. Rewriting the model with a different attention backend, hand-tuning CUDA graph capture sizes, or switching executors might give a few percent, and they all have a recurring cost in maintenance and risk on upgrade. Earn those with a measured bottleneck first, and prefer the automatic choices until a profile says otherwise.

Four errors in depth

The table gives the fix. These four are worth understanding, because they come back in new disguises.

Preemption that nobody notices

Nothing fails, so nothing alerts. Requests are evicted and recomputed, so the user sees a pause in the stream and a late finish, and your average latency barely moves because only the unlucky requests are affected. The p99 is where it appears. The cause is always the same sum: concurrent requests times their lengths exceeds the KV capacity. You can attack either side. On the supply side, raise --gpu-memory-utilization a little (it has a ceiling), switch to --kv-cache-dtype fp8, quantize weights to free memory, or add tensor parallelism. On the demand side, lower --max-num-seqs, shorten --max-model-len, or cap max_tokens at the gateway. The wrong move is adding replicas without knowing which side you were on, because each replica has the same ratio.

The quality change after an upgrade

Teams lose days to this because they suspect the model. Check what the server added on its own: the sampling defaults inherited from generation_config.json, a chat template that came with a newer tokenizer, and a changed default context length. Start by sending the same request to the old and the new version with every sampling parameter explicit, at temperature zero. If the outputs now match, the difference was defaults. If they still differ, the cause is in the weights, the template or the kernels, and you narrow it down by changing one at a time. A saved regression set turns this from an investigation into a diff.

The pod that never finishes loading

The model is fine and the image is fine; Kubernetes is killing the container. A liveness probe on /health starts failing during the minutes of weight loading and compilation, the kubelet restarts the container, and the cycle repeats forever. The log line KeyboardInterrupt: terminated is the container receiving a termination signal. The fix is a startupProbe that tolerates the slowest cold start you have measured, plus persistent caches so cold starts get shorter. Measure a cold start on the real node and size the probe from that number with generous margin.

The tool call that is almost right

Tool calling fails in quieter ways than structured output. The model emits a call, the parser extracts it, and a required argument is missing or has the wrong type. Three habits help. Validate arguments on your side before executing anything. Lower the temperature and set it explicitly. And where the argument shape is fixed, use structured outputs to constrain it. If the failure rate stays high, the model is probably too small for the task, and no flag will fix that.

Mistakes that look like tuning

Some changes feel like optimisation and are not. Raising --gpu-memory-utilization to 0.99 leaves no headroom for activation spikes and for other processes, so it trades a slow failure for a sudden one. Disabling prefix caching to "save memory" removes the cheapest speedup you have. Setting a huge --max-model-len "just in case" forces the startup check to demand KV for one enormous request, and limits what else fits. Using --enforce-eager in production because it starts faster costs steady-state throughput on every request. Leaving latest in a manifest feels convenient until the Tuesday it pulls a release that removed a flag. In each case the cheap, right alternative is to measure first and change one thing.

Practice cards

Short drills that take under fifteen minutes each. Do them on a test deployment, not production.

Habits worth adopting

Read the startup log every time you change a flag. It states the KV cache memory and capacity, the compile steps and the chosen configuration, and it is the quickest feedback loop you have. Copy the capacity numbers to your runbook per model and GPU.

Change one variable at a time and keep the benchmark output. The tuning space is large and the interactions are real: raising the token budget can lower TTFT and raise ITL, which looks like a regression if you also changed the queue limit. A saved run from before is the only way to attribute a change.

Treat deprecation warnings as tickets. Each names a removal version, and a fortnight release cadence means that version is closer than it looks. Grep startup logs in CI.

Keep prompts and logs in mind as data. Request logging is off by default; when you enable --enable-log-requests, you are writing user prompts into your log pipeline. In regulated settings, including data-residency commitments common to Gulf and Egyptian employers, decide where those logs live and for how long.

Operational recipes

Recipe: find out why latency spiked

Recipe: benchmark a configuration change

# baseline
vllm bench serve --model Qwen/Qwen3-0.6B \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --num-prompts 500 | tee baseline.txt

# change exactly one thing in config.yaml, restart, then:
vllm bench serve --model Qwen/Qwen3-0.6B \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --num-prompts 500 | tee candidate.txt

diff baseline.txt candidate.txt

Compare p95 and p99 latency and total throughput, not the mean. Run each three times if the numbers are noisy.

Recipe: a safe config file

model: Qwen/Qwen3-0.6B
revision: "<commit-sha-of-the-checkpoint>"
host: "127.0.0.1"
port: 8000
served-model-name: assistant
max-model-len: 16384
gpu-memory-utilization: 0.9
generation-config: vllm

Notes: the host is explicit, so nothing binds to every interface by accident; the revision is pinned; the served name is stable for clients; generation-config: vllm removes the checkpoint's sampling defaults, so requests must set their own parameters.

Recipe: smoke test after an upgrade

set -euo pipefail
vllm --version                               # expect the pinned version
vllm serve Qwen/Qwen3-0.6B --max-model-len 2048 &
SERVER=$!
until curl -sf http://localhost:8000/health; do sleep 5; done
curl -s http://localhost:8000/v1/models | grep -q Qwen3
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":8}'
kill $SERVER

Recipe: a minimal alert set

1. Waiting requestsAbove a threshold for five minutes. The queue is the earliest user-visible symptom.
2. Preemption rateAny sustained non-zero rate on vllm:num_preemptions. Memory is too small for the load.
3. KV usageNear full for a long window, to warn before the preemptions begin.
4. p99 TTFT and ITLAgainst the targets you agreed with product, not against a number from a blog.
5. Cache hit ratioA drop after a deploy usually means a prompt template changed.
6. Pod restartsEach restart is minutes of reduced capacity and a cold cache.

Reading the startup log

The startup log is the cheapest diagnostic you have, and most teams scroll past it. Read it in four passes. First, the version and the resolved configuration: confirm the model, the dtype, the maximum model length and whether your config file's values or a command-line override won. Second, the model load: how long it took, and whether quantization was detected from the checkpoint. Third, the memory profile: the KV cache size in gigabytes and in tokens, which is the number to compare with your arithmetic. Fourth, compilation and CUDA graph capture, which tells you whether the compile cache was reused; a long compile on every restart means the cache directory is not persisted.

Warnings deserve the same attention as errors. A deprecation warning names the version that will remove a feature. A warning about a fallback, such as a different attention backend than you expected, tells you the fast path was not available on this hardware. A warning that a generation config was applied tells you sampling defaults came from the checkpoint. Save one good startup log per environment in the runbook, next to the capacity figures and the benchmark baseline, and diff it against the next one after any change. Most "it behaves differently" reports are visible in that diff before any user notices.

When to ask for help

Gather before you ask. The output of vllm collect-env, the exact version, the full launch command or config file, the first error and the lines around it, and a reproduction on the smallest model that shows the problem. Try --enforce-eager to say whether CUDA graphs are involved. Note whether the problem appears with one GPU as well as several. With that evidence, an issue on the project's repository is actionable; without it, it waits. Remove tokens and customer data from anything you paste, including the Hugging Face token that may appear in a copied environment listing.

A short checklist

Neighbouring guides that pair with this one: Docker for the image mechanics, Kubernetes for the Deployment objects, LiteLLM for the gateway, Prometheus and Grafana for the dashboards, and Langfuse for application-level traces.

This is part three of three. The first two levels were about making vLLM serve a model well. This one is about being the person whose name is on the pager when forty other engineers depend on that endpoint: what breaks first, what a tenant can do to another tenant, what an upgrade can silently change, and when the honest answer is that vLLM is the wrong tool.

Where this picks up

You can already run vllm serve, size --max-model-len, read the KV-cache log line, and enable tool calling and LoRA. The senior question is different: not "does it work" but "what does it cost, who can abuse it, and what happens at the second replica, the tenth model, and the fifteenth release".

The version this guide is verified against is vLLM 0.30.0, released on 22 September 2026. vLLM ships a minor release roughly every two weeks, and many blog posts from 2024 and 2025 show flags that no longer exist, so treat any command you did not verify against the version you run as a hypothesis.

The process architecture and what it implies

A running vllm serve is not one process. It is a small distributed system, and every operational behaviour you will be asked to explain follows from its shape. There are API server processes that handle HTTP, tokenization, multimodal loading and detokenization. There are engine core processes, one per data-parallel rank, each running a busy loop that holds the scheduler and the KV-cache manager. And there are GPU workers, one per GPU, each containing a model runner that prepares inputs and replays captured CUDA graphs. The API servers talk to the engine cores over ZMQ; the workers talk to each other over NCCL and torch.distributed. When data parallelism is greater than one there is also a coordinator that balances load across ranks and synchronizes mixture-of-experts forward passes.

The process count is worth being able to compute, because it explains memory use and debugging output: API servers equal the data-parallel size, engine cores equal the data-parallel size, workers equal tensor parallel times pipeline parallel times data parallel, plus one coordinator when data parallelism exceeds one. A tensor-parallel-4 deployment is six processes. Tensor parallel 2 with data parallel 4 is seventeen. When someone asks why a pod shows a dozen Python processes, this arithmetic is the answer.

Three consequences matter for a platform owner. First, the unit of failure is the whole instance. A worker that dies takes the engine with it, and the in-flight KV cache and every streaming response go too. There is no partial degradation inside one vllm serve, so your resilience comes from replicas, not from the process. Second, the instance is stateful in a way that stateless web services are not: it holds a warm prefix cache, and restarting it throws away the cache that makes it fast. Treat a replica as a pet whose warm-up you pay for, even though you deploy it like cattle. Third, the API server can become the bottleneck before the GPU does. Tokenization, multimodal preprocessing and streaming serialization are CPU work in the API server processes. When GPU utilization is low and latency is high with long prompts or many images, --api-server-count is the knob, and vllm:num_requests_waiting rising while the GPU is idle is the symptom.

Startup is the other architectural fact that shapes operations. Between container start and the first successful request there is a model download or load, torch.compile with Dynamo and Inductor, and CUDA-graph capture. For a large model this takes minutes, and the engine start timeout is VLLM_ENGINE_READY_TIMEOUT_S, which defaults to 600 seconds. The consequences are concrete: a Kubernetes liveness probe that starts at once will kill pods during load (the log line is KeyboardInterrupt: terminated), compile artifacts live in VLLM_CACHE_ROOT and are lost with the container unless you persist that directory, and rolling out a new version costs a real capacity dip rather than a blip. Version 0.30.0 added --load-format ipc_cache and a vllm preload command, a per-GPU weight-cache daemon, aimed at exactly this restart cost. I have not seen enough production reports to recommend building a design around it; evaluate it on your own hardware before trusting it.

Capacity: the KV cache is the budget

At junior levels the KV cache is something the log line mentions. At this level it is the budget your platform spends. Concurrency is bounded by how many tokens of attention keys and values fit in the memory left after weights and activations, and that number depends on the model's layers, KV heads, head dimension and the cache dtype. The practical formula is simple: tokens in flight are roughly the KV memory divided by the per-token KV size. Everything else, including the number of tenants a replica can carry, falls out of that.

gpu_memory_utilization defaults to 0.92 in 0.30.0, not the 0.9 that older material states, and it is a per-instance cap on weights plus activations plus cache as a share of the total device memory. This is the source of the most common platform-sharing mistake. It is not "extra" memory and it is not a share of what is free. Two instances on one GPU each need a value near 0.45, and a notebook or a zombie process holding memory produces the startup error that begins Free memory on device cuda:0 ... on startup is less than desired GPU memory utilization. If you offer GPU sharing to other teams, the fix is a convention enforced by the platform, not a flag you hope people set. There is also --kv-cache-memory-bytes to set the cache size explicitly, which is the better primitive when you want a deterministic budget independent of what else is on the card.

The second budget is the scheduler. Each step the engine fills a token budget, max_num_batched_tokens, and caps concurrent sequences with max_num_seqs. The defaults are hardware dependent in 0.30.0 and worth knowing exactly, because "we left it at the default" gives different behaviour on different node pools. On a GPU of 160 GB or more, such as a B200, both offline and server use 1024 sequences and 16384 tokens. On a GPU of 70 GB or more that is not an A100, such as an H100 or H200, sequences are 1024 and tokens are 16384 offline but 8192 for the server. On everything else, including A100, L4 and consumer cards, it is 256 sequences and 8192 tokens offline but 2048 for the server. A fleet that mixes A100 and H100 pools therefore runs different scheduling policy per pool unless you set these explicitly. A smaller token budget favours inter-token latency; a larger one favours time to first token and throughput. --performance-mode accepts balanced (the default), interactivity and throughput, and is a coarser way to say the same thing.

Preemption is the signal that the budget is spent. When the cache runs out mid-generation, V1 evicts a request and later recomputes it. You will not see an error. You will see tail latency climb and vllm:num_preemptions increment, and since 0.30.0 a vllm:request_num_preemptions histogram that shows how many times individual requests were hit. A platform that alerts on preemption rate catches saturation before users file tickets. The remedies in order of preference are lowering --max-num-seqs or the token budget, raising capacity through an FP8 KV cache (--kv-cache-dtype fp8), quantized weights, or more tensor parallelism, and finally adding replicas.

Admission control closes the loop. Without it a saturated replica queues without bound and every request eventually times out at the client, which is worse than refusing some. Since 0.29.0 there are --max-num-queued-reqs and --max-num-queued-tokens, which let the server reject new work when the queue is full. Combine them with a gateway that returns a clear retryable status, and set VLLM_MAX_N_SEQUENCES (default 16384) lower on shared endpoints so one request cannot ask for thousands of parallel samples.

Scaling: which component breaks first

Scaling vLLM is a question of ordering. The components fail in a predictable sequence, and a platform owner who knows the sequence knows what to monitor before it happens.

The first thing to break on a single replica is the KV cache, long before compute. Decode is memory-bandwidth-bound, prefill is compute-bound, and the cache is shared between all concurrent requests, so a handful of long-context requests can crowd out dozens of short ones. You see preemption and queue time rise while GPU utilization looks healthy, which misleads people who watch only utilization.

The second is the API server's CPU. Tokenization, chat-template rendering, multimodal decoding and streaming are single-process work per API server. Large images, long documents and many concurrent streams saturate it. Add --api-server-count and watch CPU on the pod, not the GPU.

The third is the interconnect. Tensor parallelism performs all-reduce on every layer, so it scales within an NVLink domain and degrades sharply across nodes without RDMA. The tensor-parallel size must also divide the number of attention heads, which restricts the sizes you can choose. Since 0.29.0 FlashInfer all-reduce is on by default for tensor parallelism on CUDA, with VLLM_ALLREDUCE_USE_FLASHINFER=0 as the opt-out, so a regression in tensor-parallel throughput after an upgrade has an obvious first suspect.

The fourth is the router. Round-robin across replicas destroys the prefix-cache hit rate, because a conversation's next turn lands on a replica that has never seen its prefix. Past a handful of replicas, cache-aware routing stops being an optimization and becomes the difference between paying for prefill once and paying for it every request.

The fifth, for mixture-of-experts models, is expert load. Experts are not used evenly, so the busiest rank sets the pace. --enable-expert-parallel shards the experts, and --enable-eplb with --eplb-config rebalances them, but this is the most specialised corner of the engine and the one where the documentation moves fastest.

Parallelism and topology decisions

The documentation's rule of thumb is the one to adopt as policy: tensor parallel size equals the GPUs per node, and pipeline parallel size equals the number of nodes. Tensor parallelism splits every layer and needs the fastest links you have; pipeline parallelism splits layers between stages and moves much less data, so it tolerates slower links. Data parallelism replicates the model and is the right tool when the model fits and you need more throughput.

# one node, four GPUs
vllm serve <model> --tensor-parallel-size 4

# two nodes of eight GPUs, multiprocessing backend, no Ray
vllm serve /path --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 0 --master-addr <HEAD_IP>
vllm serve /path --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 1 --master-addr <HEAD_IP> --headless

The alternative is Ray, with --distributed-executor-backend ray and the cluster script from the examples directory. Choose by operational ownership. If your organisation already runs Ray and has people who can debug it, the integration is natural, and the Ray guide in this catalogue (Ray) covers the platform side. If you do not, the multiprocessing multi-node path removes a moving part. Whichever you pick, verify the network instead of assuming it. Run the documented sanity script under NCCL_DEBUG=TRACE and look for [send] via NET/IB/GDRDMA, which is the good path, versus NET/Socket, which means collective traffic is going over plain TCP and your multi-node tensor parallelism will be slow.

For data parallelism the engine offers three load-balancing arrangements, and the choice is an architecture decision. Internal balancing gives one endpoint and lets the coordinator spread requests across ranks. Hybrid balancing runs API servers per node and expects an upstream balancer to spread across nodes. External balancing treats each rank as an independent server and leaves all balancing to you. Internal is the simplest and the first to hit a single-endpoint ceiling; external is the most flexible and the most work. The flags are --data-parallel-size, --data-parallel-size-local, --data-parallel-hybrid-lb and --data-parallel-external-lb, with --headless for worker nodes.

Routing, replicas and the gateway

A single replica is a pet; a pool of replicas needs a router, and the router is where a platform adds most of its value. The engine itself offers no per-tenant accounting, rate limiting, budgets or key management. Those belong to a layer you own, and the open question is only which one.

You have three realistic choices. A general gateway such as LiteLLM (LiteLLM) gives virtual keys, budgets, retries and usage logs across vLLM and hosted providers, which suits platforms that must front many backends. An ingress or proxy layer such as NGINX or Envoy is the right place for authentication, endpoint allowlisting and TLS, as the next sections explain. And the production-stack project, listed in the official documentation, provides a router with round-robin, session, prefix-aware, KV-aware, load-aware and disaggregated-prefill routing for Kubernetes. Other projects the documentation lists for Kubernetes include llm-d, KServe (KServe), AIBrix, NVIDIA Dynamo, KubeRay and LeaderWorkerSet for multi-node placement. I would not claim to rank them without running them against your traffic; the stable evaluation criteria are whether the router understands prefix and KV state, how it handles a replica that is warming up, and who maintains it.

The engine helps cache-aware routers with --kv-events-config, which publishes KV block events so a router can learn which replica holds which prefix. That is the mechanism behind KV-aware routing, and it is the reason to prefer a router that consumes those events over one that guesses from request hashes. Session affinity is a cruder version that gets part of the benefit: pin a conversation to a replica by header and accept uneven load.

Health checking deserves care. /health returns 200 when the server is ready, but readiness at the router should also consider load. The /load endpoint exists for that purpose, and autoscaling should key on vllm:num_requests_waiting and KV-cache usage rather than CPU, which stays low on an inference pod even when it is saturated. A readiness probe that passes during warm-up would send traffic to a replica that has not captured CUDA graphs; use a startup probe with a generous failure threshold and let readiness begin only after startup succeeds.

Security: the trust model

The most important security fact about vLLM is that its authentication is narrower than it looks. The --api-key flag guards the /v1, /v2 and /inference prefixes and the Cohere-compatible paths. It does not guard everything the server exposes. The help text itself warns that /invocations, the SageMaker-compatible route, remains unauthenticated, and the official security guidance names other endpoints such as /tokenize, /pause, /resume, /abort_requests and /update_weights as outside that protection. The guidance is blunt: do not rely on --api-key alone. For a platform that means vLLM is never exposed directly. It sits behind a reverse proxy or API gateway that authenticates callers and allowlists the specific paths your product exposes, and denies the rest by default.

The same discipline applies to the surfaces that turn vLLM from a server into a remote-code-execution target. VLLM_SERVER_DEV_MODE=1 exposes /collective_rpc, which is arbitrary RPC to the workers, alongside the sleep and cache-reset endpoints. --enable-tokenizer-info-endpoint, --profiler-config and runtime LoRA updating through VLLM_ALLOW_RUNTIME_LORA_UPDATING are development conveniences with security cost. The platform default should be that none of these are enabled, with a policy check in CI that fails a deployment carrying them. --trust-remote-code executes Python from the model repository at load time, so it is only for repositories you have reviewed, ideally pinned to a commit with --revision. --trust-request-chat-template, default off, lets a client send its own chat template, which belongs on only for clients you trust.

Network exposure is the second half. Inter-node traffic is insecure by default: torch.distributed, KV transfer (which defaults to 127.0.0.1 port 14579 through --kv-ip and --kv-port) and data-parallel coordination are unauthenticated and unencrypted. The --grpc server is also unauthenticated. The answer is network isolation, not configuration: a private subnet or Kubernetes network policy for the engine pods, VLLM_HOST_IP set to the interface you intend, and a firewall that allows only the router to reach the API port. /metrics is unauthenticated as well, so scrape it from inside the cluster and never route it publicly. CORS defaults to --allowed-origins ['*'], which you should tighten, and TLS is either --ssl-keyfile and --ssl-certfile or terminated at the proxy.

Multimodal input adds server-side request forgery risk, because the server fetches URLs that clients supply. Constrain it with --allowed-media-domains, --allowed-local-media-path, VLLM_MEDIA_URL_ALLOW_REDIRECTS=0, and the size and pixel limits VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB (default 256), VLLM_MAX_IMAGE_PIXELS and VLLM_MAX_AUDIO_CLIP_FILESIZE_MB (default 25). The latest documentation also describes keeping per-request multimodal processor arguments disabled unless an explicit trust flag is set; that flag is not in the 0.30.0 source, so treat it as upcoming and check your version before relying on it.

Secrets and the supply chain

Two secrets matter: the Hugging Face token for gated models and the API key. Deliver both through your secret manager or a Kubernetes Secret via secretKeyRef, and prefer the VLLM_API_KEY environment variable to the --api-key flag, because command-line arguments are visible to anyone who can run ps on the node. Never bake either into an image. If your platform brokers keys per tenant, those belong at the gateway, and the engine's own key becomes a single shared secret between router and replicas. The Vault guide in this catalogue (Vault) covers dynamic secret injection if you run it.

The supply chain has more links than the engine. There is the vLLM wheel or image, which is pinned against a specific PyTorch (2.13.0 in 0.30.0), so installing a different torch afterwards breaks the compiled kernels with undefined-symbol errors on vllm._C. There is the CUDA runtime and the driver, where a driver older than the wheel's CUDA produces the provided PTX was compiled with an unsupported toolchain. There are model weights, which are a code and data artifact you download from the outside world, and the quantized checkpoints derived from them. And there are the plugins, since bitsandbytes became an out-of-tree plugin in 0.28.0 and tool parsers can be loaded with --tool-parser-plugin.

The controls are the ones from the Docker guide (Docker), applied with GPU-specific care. Pin the image to an exact tag such as vllm/vllm-openai:v0.30.0, and better, to its digest, never latest. Build your own derived image from that base when you need extra dependencies, scan it, and promote it through environments. Run as non-root: the official image has a built-in vllm user with UID 2000 and group 0, so --user 2000:0 works and caches belong under /home/vllm/.cache. Mount model weights from a controlled store rather than pulling from the public hub at pod start, which removes both a startup dependency and an exposure to repository changes. Pin --revision for anything pulled by name. For weights, record the checksum or commit you evaluated, so that an incident review can answer which exact weights served a request.

Multi-tenancy and isolation

Sharing a replica between tenants is the economically attractive choice and the one that needs the most thought, because the engine was designed for throughput, not for isolation between mutually distrusting callers. What it offers is a set of tools, and the platform composes them into a policy.

The first tool is LoRA multiplexing. Instead of one deployment per fine-tune, a single base model serves many adapters, selected by the model field in each request. --enable-lora, --lora-modules name=path, --max-loras (default 1) and --max-lora-rank (default 16) configure it. For a team that fine-tunes per customer this changes the cost shape from linear in tenants to near-constant. The caution is runtime loading: the /v1/load_lora_adapter endpoint requires VLLM_ALLOW_RUNTIME_LORA_UPDATING, and exposing it to tenants lets them load arbitrary files into your engine. Keep adapter registration an internal, reviewed operation, and use the resolver plugins or a rollout of config if you need it automated.

The second tool is prefix-cache isolation. Automatic prefix caching is on by default and its block hash includes the parent hash, the tokens and extra hashes such as the LoRA identifier. The official security guidance notes that timing differences can reveal whether another tenant already sent a given prompt. When one tenant could infer another's system prompt or document, give each tenant a cache_salt in the request, set at the gateway so tenants cannot choose it, which makes their blocks hash differently. It costs you cross-tenant cache sharing, which is the point. Multimodal UUIDs should likewise be random.

The third is priority. --scheduling-policy priority with a per-request priority lets the scheduler favour premium or interactive traffic over batch. It does not give quotas. A tenant that sends enormous requests still consumes cache and preempts others' work, so quota enforcement (requests per minute, tokens per minute, a cap on max_tokens) stays in the gateway. The engine has no per-user accounting at all, which is why the gateway owns keys, budgets and usage logs.

The fourth is physical separation, which remains the strongest isolation. Run a dedicated pool for tenants whose data classification forbids sharing, and a shared pool for the rest. This is more expensive and much easier to explain to an auditor. A reasonable platform offers both tiers and prices them differently.

Cost: where the money actually goes

Inference cost is GPU-hours divided by useful tokens, and a platform owner has levers on both sides. The denominator is the one people neglect. A GPU that serves at low utilization because the router destroys cache hits, because max_model_len is generous, or because every product has its own replica is a GPU wasted, and no quantization will fix an architecture problem.

The first lever is quantization, which shrinks weights and the KV cache so more fits per GPU. FP8 is the common starting point on hardware that supports it, with AWQ, GPTQ, NVFP4 and compressed-tensors checkpoints produced by tools like llm-compressor as alternatives, and --kv-cache-dtype fp8 for the cache. Be careful about the version notes: GPTQ checkpoints that depend on activation ordering stopped loading in 0.30.0, which matters if your model catalogue contains older community quantizations. Quantization trades quality, so evaluation is part of the change and not a follow-up.

The second lever is reuse: prefix caching with cache-aware routing, discussed above, and LoRA multiplexing instead of one deployment per fine-tune. The third is right-sizing --max-model-len and the scheduler limits to the real traffic, since both determine how many requests a replica holds. The fourth is speculative decoding, which buys latency and sometimes cost, but only under conditions you must measure.

Measure with vllm bench serve, which accepts a dataset, input and output lengths and a prompt count, and report goodput: the throughput that meets your latency objective, not raw tokens per second. Raw throughput can be bought by letting latency explode, and a cost model that ignores the objective rewards the wrong configuration. Region matters here too. If your data-residency rules keep traffic inside a Gulf or Egyptian region, GPU availability and price there are part of the cost model, and a self-hosted engine on in-region capacity is often the reason a team chose vLLM over a hosted API in the first place.

Observability and service objectives

Define the objectives before the dashboards. For interactive products the useful ones are time to first token (vllm:time_to_first_token_seconds), inter-token latency (vllm:inter_token_latency_seconds) and end-to-end latency (vllm:e2e_request_latency_seconds), reported as percentiles per model and per tenant. For batch products it is throughput and completion time. Prefill is compute-bound and drives first-token latency; decode is memory-bandwidth-bound and drives inter-token latency, so a regression in one points at a different cause than the other.

The Prometheus endpoint at /metrics exposes the series you need. For saturation: vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc and vllm:request_queue_time_seconds. For phases: vllm:request_prefill_time_seconds and vllm:request_decode_time_seconds. For volume: vllm:prompt_tokens and vllm:generation_tokens. For cache health: vllm:prefix_cache_hits and vllm:prefix_cache_queries. For trouble: vllm:num_preemptions, vllm:request_success and vllm:corrupted_requests. Scrape them with Prometheus (Prometheus) and chart them in Grafana (Grafana).

groups:
  - name: vllm
    rules:
      - alert: VllmQueueGrowing
        expr: avg_over_time(vllm:num_requests_waiting[5m]) > 5
        for: 10m
        labels: { severity: page }
      - alert: VllmKvCachePressure
        expr: max_over_time(vllm:kv_cache_usage_perc[10m]) > 0.95
        for: 15m
        labels: { severity: ticket }
      - alert: VllmPreempting
        expr: rate(vllm:num_preemptions[10m]) > 0
        for: 15m
        labels: { severity: ticket }

Treat the thresholds as starting values to calibrate, and verify the exact metric names against the /metrics output of the version you run, since deprecated series are hidden one minor release later and can be re-shown with --show-hidden-metrics-for-version. That flag is a migration aid for dashboards, not a long-term setting.

Upgrades and migrations

vLLM's deprecation policy is explicit and your upgrade process should mirror it. Versions are semantic, and removals happen on minor releases. A deprecated item first still works with a warning naming the removal version, then is off by default and re-enabled by a flag, then is removed. The warnings are your early notice, so a platform should collect engine logs and alert on the string "deprecated" in a staging environment long before a removal lands.

The recent history shows what the failures look like. In 0.29.0, Model Runner V2 became the default for all models, with the V1 runner deprecated and removal targeted for 0.32, and the entrypoint python -m vllm.entrypoints.openai.api_server was deprecated in favour of vllm serve. Ten model architectures were removed, including MPT, Arctic and Chameleon. In 0.30.0 the scale-out endpoints became opt-in with --enable-scale-out and their environment variable was removed, the standalone gRPC entrypoint was deprecated in favour of vllm serve --grpc, GPTQ activation ordering support was removed, several environment variables became CLI flags, and the ROCm fallback from CUDA_VISIBLE_DEVICES was removed in favour of HIP_VISIBLE_DEVICES. Earlier removals still catch teams on old configuration: the V0 engine and VLLM_USE_V1, VLLM_ATTENTION_BACKEND (use --attention-backend), --task (use --runner and --convert), --swap-space, and the guided_* request fields (now structured_outputs).

The most dangerous changes are the silent ones. Sampling defaults come from the model's generation_config.json since 0.8.0, so an upgrade or a model swap can change temperature behaviour without an error, and "quality changed" tickets are often this; --generation-config vllm restores the engine's own defaults, and the platform should choose and document one policy. Default context length can change when the model-configuration handling changes. The logprobs semantics are another trap: by default V1 returns raw logprobs before temperature and penalties, and --logprobs-mode changes it. Each of these passes a smoke test and fails an evaluation, which is why the evaluation belongs in the pipeline.

Mechanically, build the new environment from a clean virtual environment or image, because vLLM pins its PyTorch, and pin the version exactly with uv pip install vllm==0.30.0 --torch-backend=auto. Roll out with a canary or blue-green, since restarting a replica discards its in-flight requests and warm cache, and drain gracefully; --shutdown-timeout exists for that purpose. Keep the previous image available for rollback and keep the previous version's configuration under version control next to it. Define a cadence: given a two-week release rhythm, adopting every release is a full-time job, so many platforms track every second or third minor release and take security fixes immediately. Pick one, publish it, and communicate removals to tenants before they happen.

Governance: who decides what

A platform without decision rights becomes a support queue. Governance for vLLM is mostly a short list of things that need an owner and a record, and the order matters because the early ones prevent the later ones.

Model admission is the first. Decide what may be served, by whom, from where. A reasonable policy: models enter through a request that records the source repository and pinned revision, the licence, the evaluation results on the use case, the hardware profile with its measured capacity, and whether --trust-remote-code is required (a reason to escalate to a security review). Weights land in your own storage, and the deployment references that, not the public hub. Licences deserve a line because several popular model licences restrict use or redistribution, and the platform team is the one place that sees all deployments.

Configuration policy is the second. Express the defaults and the forbidden settings as code: a Helm chart (Helm) or Kustomize base with the safe flags, plus an admission check that rejects pods carrying VLLM_SERVER_DEV_MODE, runtime LoRA updating, --trust-remote-code without an exception, an unpinned image tag, or a host binding exposed outside the mesh. Tenants override model-level parameters such as --max-model-len and quantization within bounds; they do not override security flags. Argo CD (Argo CD) reconciles the result, so a manual change is reverted rather than remembered. Provisioning the underlying GPU pools with Terraform (Terraform) keeps capacity changes reviewable.

Change control is the third: the upgrade cadence above, a registry of which model runs on which version, and an exception process with expiry dates. Data governance is the fourth: where prompts are logged, how long they are retained, who can read them, and which regions they may leave. For Gulf and Egyptian employers this is often the hardest question and the one that decides whether the engine runs in a local region or a hosted API is acceptable.

Incident playbooks

Write these before you need them; each is short, and the value is in knowing the first command. Throughout, the first move in nearly every case is to check the engine's own log, because vLLM reports a lot, and to run vllm collect-env when you are not sure what is installed.

Replica will not start. Scroll above the wrapper. RuntimeError: Engine core initialization failed. See root cause above is never the cause; the real error sits one to three lines up. The common causes are memory errors (the KV-cache ValueError that names an estimated maximum length, the No available memory for the cache blocks message, or the Free memory on device message when another process holds the GPU), authentication or download failures for a gated model, an unsupported architecture after an upgrade, Bfloat16 is only supported on GPUs with compute capability of at least 8.0 on a T4 (use --dtype=half), and CUDA toolchain mismatch. For the memory ones, the log's own suggestion is correct: lower --max-model-len, raise utilization if nothing else is on the card, or quantize.

Pod restarts forever during load. If the log ends with KeyboardInterrupt: terminated, the orchestrator killed it. The startup probe or liveness probe fired before weights finished loading. Raise the failure threshold, use a startup probe, and persist the cache. A related trap: Kubernetes injects environment variables for every Service in a namespace, so a Service named vllm injects VLLM_PORT=tcp://..., which collides with vLLM's own internal setting. Rename the Service.

Latency climbs, GPU looks fine. Check vllm:num_requests_waiting, vllm:kv_cache_usage_perc and vllm:num_preemptions. Rising preemption means the cache is full; reduce concurrency limits, shed load with the admission flags, or add replicas. If waiting is high while the GPU is idle, suspect the API server CPU and raise --api-server-count. If one tenant's requests are long, a gateway cap is the fix.

Multi-GPU hang or NCCL failure. NCCL error: unhandled system error at communicator initialisation points to a missing shared-memory mount or capability: --ipc=host or --shm-size for Docker, a memory-backed emptyDir at /dev/shm in Kubernetes, IPC_LOCK for RDMA. On PCIe machines Cuda failure 217 'peer access not supported' is addressed with NCCL_CUMEM_HOST_ENABLE=0 or a driver upgrade, and NCCL_P2P_DISABLE=1 is a diagnostic workaround, not a fix. Run the sanity script with NCCL_DEBUG=TRACE before blaming vLLM. Multi-node hangs are usually the wrong network interface: set VLLM_HOST_IP, NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME.

Quality or behaviour changed after a deploy. Compare the version, the model revision and the generation defaults. Ask whether generation_config.json is now being applied, whether the default context length changed, whether a quantization changed, and whether a chat template or tool parser changed. Reproduce with --generation-config vllm to separate engine defaults from model defaults.

Suspected abuse or exposure. Treat an exposed engine as compromised until proven otherwise: find the reachable paths, disable dev-mode and runtime LoRA endpoints, rotate the Hugging Face token and API key, and review --trust-remote-code models. Then close the gap at the gateway. Report any vLLM vulnerability through the project's security policy on GitHub.

For a crash you cannot explain, --enforce-eager tells you whether CUDA graphs are involved, VLLM_LOGGING_LEVEL=DEBUG adds detail, CUDA_LAUNCH_BLOCKING=1 makes CUDA errors synchronous and attributable, and VLLM_ENABLE_V1_MULTIPROCESSING=0 runs in-process so a debugger can attach. VLLM_TRACE_FUNCTION=1 exists but slows execution by more than a hundred times, so reserve it for an isolated reproduction.

Where vLLM stops

A platform owner earns trust by saying no. vLLM is an excellent general engine for serving open-weight transformer and mixture-of-experts models, and there are real cases where something else is better.

If you need the last increment of performance on NVIDIA hardware and can accept the build and compile effort, TensorRT-LLM (TensorRT-LLM) is the vendor-optimised alternative, with an engine-build step vLLM does not require. If your workload is dominated by structured, multi-call programs with heavy shared prefixes, SGLang (SGLang) is designed around that pattern and is worth benchmarking against vLLM on your own traces. Text Generation Inference (TGI) is another server with a similar role. I would not declare a winner between these in the abstract. The honest procedure is to benchmark two candidates on your model, hardware and request distribution with vllm bench serve and the other tool's equivalent, and to include operational cost, not just throughput.

If you need a general model-serving platform with many frameworks, versioning and multi-model management, vLLM is a backend and not the platform. Triton Inference Server has a documented vLLM backend (Triton), BentoML (BentoML) can wrap it, and KServe (KServe) runs it on Kubernetes. If you need a single developer's laptop experience, Ollama and llama.cpp are simpler, and vLLM on Apple silicon is an experimental CPU backend built from source, with Metal acceleration only through a community plugin. Windows has no native support and runs through WSL2.

vLLM also does not give you what a managed provider gives: elastic capacity without owning GPUs, a vendor accountable for model updates, and compliance certifications. Self-hosting wins when volume is high and steady, when data must stay in region, when you need adapters or custom models, or when per-token cost at your scale clearly beats an API. It loses when traffic is spiky and small, when the team cannot staff an on-call rotation for GPUs, or when the model you want is only available through a hosted API. That breakeven deserves an actual spreadsheet with GPU-hours, utilization and engineering time in it, which you revisit when prices change.

Running vLLM as a platform for other teams

Everything above converges here. The deliverable is not a running server but an internal product, with users who have expectations, and the product has a surface, a contract and a support model.

The surface is the OpenAI-compatible API behind your gateway, because every client library, framework and tool already speaks it. Tenants use their normal SDK with a different base URL. Keep that promise stable: the platform, not tenants, absorbs the engine's flag churn. Version the platform interface separately from the engine version, so that an upgrade of vLLM is not a breaking change for the product.

The contract states what is guaranteed. For each model tier, it lists the context length, the supported features (tool calling and its parser, reasoning parser, structured outputs, multimodal limits, LoRA), the rate and token limits, the latency objectives measured at the gateway, the maintenance windows, and the deprecation notice period. A tenant who needs a feature outside the contract opens a request, and that is how model admission and capacity planning get fed. Document the sharp edges honestly: sampling defaults come from the model, long contexts are capped, and structured-output syntax uses structured_outputs, not the removed guided_* fields.

The support model is a runbook, an on-call rotation and a feedback loop. The runbook holds the playbooks above with the commands filled in for your cluster. On-call has a dashboard per model with the objectives, capacity and preemption rate on one page. The feedback loop is a monthly review of usage and cost per tenant, the top deprecation warnings, and the capacity forecast. Publish a self-service path for the common requests, such as a new LoRA adapter, a new model in an existing tier, and a quota change, because the platform's effective headcount is the number of requests it can handle without a human.

The review checklist

Use this when reviewing a deployment pull request or a design document. Each line should be answerable from the diff or the document.

1. Version. The vLLM version and image digest are pinned. Release notes between the old and new version were read, and removals were checked against the flags in use.
2. Exposure. The engine is reachable only from the gateway. Authentication and endpoint allowlisting happen at the gateway. /metrics and --grpc are not public.
3. Dangerous surfaces. No VLLM_SERVER_DEV_MODE, no runtime LoRA updating, no profiler configuration, no tokenizer-info endpoint. --trust-remote-code has a recorded exception and a pinned revision.
4. Secrets. Tokens come from the secret manager, and the API key is passed by environment, not argv.
5. Capacity. --max-model-len, scheduler limits and gpu_memory_utilization are set deliberately, with a measured capacity number and a preemption alert.
6. Overload. Admission limits exist and the gateway returns a retryable status. max_tokens and n are capped.
7. Topology. Tensor parallel matches GPUs per node. Multi-node traffic uses RDMA, verified with NCCL_DEBUG. Shared memory is mounted.
8. Routing. The router is cache-aware or has session affinity, with a plan for warm-up and drain.
9. Isolation. Shared pools set cache_salt at the gateway. Tenant data classification matches the pool.
10. Startup. Startup probe thresholds cover load and compile time. The compile cache persists and is private.
11. Observability. Objectives, dashboards and alerts exist for queue, KV usage, preemption and latency. Request logging has a decision about prompt content.
12. Quality. The generation-config policy is explicit. An evaluation ran on the new version.
13. Rollout. Canary or blue-green, rollback path, and a drain period.
14. Ownership. There is an owner, an on-call route and a catalogue entry.

The complete picture

Put the three levels together and a coherent picture emerges. At the base, vLLM is an engine that turns GPU memory into concurrent token generation by batching continuously, paging the KV cache and reusing shared prefixes. Around that engine sit the choices that determine whether it is cheap and safe: how the model is sharded across GPUs, how many replicas run and how requests reach them, what the gateway enforces, how tenants are isolated, and how the platform is upgraded without surprises.

The thread running through the senior level is that the engine is deliberately not a platform. It has no per-tenant accounting, only partial authentication, an unauthenticated inter-node protocol, and a release cadence that removes things. Each of those is a sensible choice for an engine and a hazard for a service, and the platform owner's job is to supply what the engine leaves out: a gateway, a network boundary, a pinning and evaluation process, an observability contract, and a set of written rules about what may be served.

Neighbouring guides cover the surrounding pieces: the Docker guide for the image and supply chain (Docker), Kubernetes (Kubernetes) and KServe (KServe) for orchestration, LiteLLM (LiteLLM) for the gateway, and Langfuse (Langfuse) for application-level tracing.

Where the series leaves you

You started by running a model and watching tokens appear. You now have what a platform owner needs: a way to predict which component fails first, a trust model that does not rely on a single flag, a multi-tenant design with named trade-offs, an upgrade process built for a fast-moving project, playbooks for the incidents that actually occur, and the judgement to say when another engine or a hosted API is the better answer.

Do three things next. Write the capacity number for each model you serve, measured and dated, and attach it to the service catalogue. Audit one live deployment against the review checklist and fix what you find, starting with exposure and dangerous surfaces. And schedule the upgrade rehearsal: pick the next release, read its breaking changes, and run the pinned canary process end to end before you need it in anger. The engine will keep changing every two weeks. A platform that expects that, and absorbs it, is the one other teams will keep using.

Sources

• vLLM documentation
• Architecture overview
• Parallelism and scaling
• Data parallel deployment
• Engine arguments
• Environment variables
• Optimization and tuning
• Conserving memory
• Security
• Metrics
• Troubleshooting
• Automatic prefix caching
• LoRA adapters
• Speculative decoding
• Disaggregated prefill
• Sleep mode
• Docker deployment
• Kubernetes deployment
• Production stack
• Deprecation policy
• Release notes
• vLLM 0.30.0 release
• Security policy

Part three of three. Senior vLLM interviews rarely ask you to recite flags. They hand you a vague business problem, a pager scenario, or a sceptical executive, and watch how you reason. This file is organised the way those conversations go: design prompts first, then incidents, then trade-off arguments, then the short answers to have ready. The version assumed throughout is vLLM 0.30.0.

How to answer at this level

Three habits separate a senior answer from a mid-level one. State your assumptions out loud, because "serve a model" has no answer until you know the model size, the context length, the traffic shape and the latency objective. Name what you would measure before what you would change. And say what you would not build yet, since restraint is the thing interviewers are listening for.

A reliable structure is requirements, capacity, topology, edge, operations, then failure. Requirements pin the objective. Capacity is KV-cache arithmetic. Topology is the smallest parallelism that fits. The edge is the gateway that authenticates and limits. Operations is observability and upgrades. Failure is what happens when a replica dies.

System design prompts

Design an internal LLM platform for forty product teams

What a strong answer covers. Begin with questions: how many models, which latency classes, which data classifications, which regions. Then propose two tiers: an interactive tier tuned for time to first token and a batch tier tuned for throughput, because one scheduler setting cannot serve both well. Put a gateway in front of the engines that owns identity, per-team keys, quotas, rate limits, a cap on max_tokens, endpoint allowlisting and usage logging, because vLLM itself has no per-tenant accounting and its --api-key guards only the /v1, /v2 and /inference prefixes. Run vLLM replicas as pinned images with a persisted private compile cache, probes that tolerate minutes of startup, and admission limits (--max-num-queued-reqs, --max-num-queued-tokens) so overload sheds load instead of queueing. Route with prefix awareness so conversations keep their cache hits. Share base models across teams with LoRA multiplexing when teams fine-tune, and give sensitive tenants a dedicated pool. Describe the contract each model tier publishes: context length, features, limits, objectives.

What a weak answer misses. It starts with GPU counts. It exposes the engine directly. It treats --api-key as a security boundary. It has no answer for how a tenant gets a new model or what happens during an upgrade.

Follow-ups to expect. How do you stop one team from starving the others? The honest answer is layered: gateway rate limits, a token cap, engine admission limits, and priority scheduling (--scheduling-policy priority) for premium traffic, with dedicated pools where isolation must be absolute. How do you charge back cost? From gateway usage logs, because the engine does not attribute usage to users.

A team wants to serve a 70-billion-parameter model with a 128k context. Size it.

What a strong answer covers. Do the arithmetic aloud. Weights in 16-bit are roughly twice the parameter count in bytes, so around 140 GB, which exceeds any single GPU and forces tensor parallelism across several, or quantization to FP8 which halves it. Then the KV cache: per-token size depends on layers, KV heads, head dimension and dtype, and a 128k context multiplies that by 128,000 for a single request. Say plainly that you would not trust a number from memory and would read the startup log, which reports the KV cache size and the maximum concurrency at that context, or the error that tells you the estimated maximum max_model_len. Conclude with levers in order: FP8 weights if quality allows, --kv-cache-dtype fp8, a smaller served context if the product does not need 128k, more tensor parallelism inside one NVLink node, then pipeline parallelism if it must span nodes. Point out that the tensor-parallel size must divide the attention-head count.

What a weak answer misses. It ignores that the cache, not the weights, limits concurrency. It forgets that a long default context fails at startup if one request cannot fit, and that serving 128k to everyone is a capacity decision with a price.

Follow-up. What about a team that insists on the full context? Make it a separate tier with separate pricing and a low concurrency expectation, rather than letting a few long requests degrade a shared pool.

Two nodes of eight GPUs, one very large model. How do you lay it out?

What a strong answer covers. The documented rule: tensor parallelism equals GPUs per node, pipeline parallelism equals nodes, so eight and two. Tensor parallelism communicates on every layer and belongs inside NVLink; pipeline parallelism moves activations between stages and tolerates the inter-node network. Mention two ways to launch it: Ray with --distributed-executor-backend ray, or the native multiprocessing path with --nnodes, --node-rank, --master-addr and --headless on the worker node. Choose between them by who in the organisation can operate Ray. Verify the fabric, do not assume it: run the sanity script with NCCL_DEBUG=TRACE and look for NET/IB/GDRDMA rather than NET/Socket. Isolate the traffic on a private network, because torch.distributed and the coordination protocols are unauthenticated and unencrypted. If the model is a mixture of experts, bring up expert parallelism (--enable-expert-parallel) and expert load balancing as a second phase.

What a weak answer misses. It chooses tensor parallelism across nodes over a slow network. It never checks which transport NCCL actually selected.

Interactive chat and a nightly batch job share one cluster. Design the scheduling.

What a strong answer covers. Do not mix them on one replica if you can avoid it, because batch work fills the KV cache and causes preemption for interactive users. Separate pools with different scheduler settings: a smaller token budget and --performance-mode interactivity for chat, a large budget and throughput for batch. If cost forces sharing, use priority scheduling with request priorities and a hard limit on batch concurrency at the gateway. For the batch itself, the offline interface or vllm run-batch with OpenAI batch-format files avoids the HTTP path entirely. Mention that sleep mode and scale-to-zero let the batch pool release GPUs outside its window, with the cold-start cost accepted because nobody is waiting.

How would you give each of fifty customers their own fine-tuned model?

What a strong answer covers. Fifty deployments is the wrong answer. Serve one base model with LoRA adapters: --enable-lora, --lora-modules, a raised --max-loras and an adequate --max-lora-rank, with tenants selecting an adapter by the model field. Control how adapters arrive: the runtime load endpoint requires VLLM_ALLOW_RUNTIME_LORA_UPDATING and lets whoever can reach it load files into the engine, so keep that off the tenant surface and make registration an internal reviewed step. Combine with per-tenant cache_salt set at the gateway so prefix-cache timing cannot reveal one customer's prompts to another. Close with the limits: adapters must match the base model and rank limits, and a customer whose data must never share a process gets a dedicated deployment.

Design for a regulated employer in Riyadh or Cairo where prompts cannot leave the country

What a strong answer covers. Self-hosting in an in-region cloud or on-premise GPU capacity, with the engine on private networking and no public hub access at runtime: weights are copied into a controlled in-region store and referenced from there, with --revision pinned. Usage-statistics telemetry is opted out with VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1. Prompt logging is a decision, not a default: request logging is off unless --enable-log-requests is set, and any logs of prompts are stored in region with a retention period. The honest trade-off is capacity: regional GPU supply and price may push toward fewer, larger quantized deployments, and the answer should say how you would verify that rather than assert it.

Incident-response questions

Latency doubled at 14:00 but GPU utilization is flat. What do you do?

Walk through the signals in order. Check vllm:num_requests_waiting and vllm:request_queue_time_seconds to see whether requests are queueing. Check vllm:kv_cache_usage_perc and vllm:num_preemptions: preemptions mean the cache is full and requests are being recomputed, which raises tail latency without raising utilization. Split latency into prefill and decode with vllm:request_prefill_time_seconds and vllm:request_decode_time_seconds; a prefill spike suggests new long prompts, a decode slowdown suggests more concurrency or a slow interconnect. Then ask what changed at 14:00: a new tenant, a longer context, a deploy, a router change that broke cache affinity (check the prefix-cache hit ratio from vllm:prefix_cache_hits over vllm:prefix_cache_queries). Mitigate before diagnosing fully: shed load with admission limits, cap max_tokens at the gateway, add replicas.

The strong answer also notes what flat utilization means. Decode is memory-bandwidth-bound, so a GPU can look busy or idle in ways that do not map to user experience, and utilization is a poor autoscaling signal compared with waiting requests and KV usage.

After Tuesday's upgrade, a team says answers feel different. Nothing errors.

This is the silent-change class. Compare version, model revision, and generation defaults. Since vLLM 0.8.0 sampling defaults come from the model's generation_config.json, so a model or version change can alter default temperature and top-p. Reproduce one request with --generation-config vllm to isolate engine defaults from model defaults. Check whether the default context length changed: 0.30.0 aligned YaRN handling with Transformers and some models' default dropped. Check the chat template and tool parser. Check whether a quantized checkpoint was swapped. If logprobs are involved, remember V1 returns raw logprobs by default and --logprobs-mode changes that. Then fix the process gap: an evaluation set that runs on every engine upgrade would have caught it.

A pod restarts in a loop. The last log line is KeyboardInterrupt: terminated.

The orchestrator killed the process, and the likeliest reason is a liveness or startup probe that fired before the model finished loading and compiling. The fix is a startup probe with a generous failure threshold and a readiness probe that begins only after it, not a larger vLLM timeout. Also check that the cache directory persists, since recompiling on every restart stretches load time. If the pod uses a Service named vllm, rename it: Kubernetes injects VLLM_* environment variables for every Service, and VLLM_PORT=tcp://... collides with vLLM's internal setting.

A security reviewer finds the engine reachable from the corporate network, --api-key set. Is that acceptable?

No, and say why concretely. The key guards only certain path prefixes, and the engine help text warns that /invocations stays unauthenticated; other paths such as /tokenize and the control endpoints are also outside that protection. /metrics is unauthenticated. Inter-node and KV-transfer traffic is plaintext. The remediation is a gateway that authenticates and allowlists paths, a network policy restricting the engine to the gateway, and a check for dev surfaces: VLLM_SERVER_DEV_MODE exposes /collective_rpc, which is arbitrary RPC to workers. Then rotate the keys, since they may have been exposed, and review which models used --trust-remote-code.

Startup fails with "Free memory on device ... is less than desired GPU memory utilization"

Something else holds GPU memory. Run nvidia-smi for stray processes: a previous instance that did not exit, a notebook, another pod on the same device. The setting is a share of total memory per instance, not of what is free, so two instances on one GPU need roughly 0.45 each. The platform lesson is to stop depending on goodwill: enforce one pod per GPU through the scheduler, or set explicit budgets with --kv-cache-memory-bytes.

A multi-node deployment hangs at startup

Work through the network. Wrong interface selection is the most common cause: set VLLM_HOST_IP, NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME. Run the documented sanity script with NCCL_DEBUG=TRACE and read the transport lines. NCCL error: unhandled system error at communicator initialisation points to missing shared memory or capabilities in the container. A model loading from a network file system can also look like a hang; copy it to local disk, and use --load-format dummy to separate loading from everything else.

Arguing trade-offs to leadership

Why self-host instead of paying a hosted API?

Lead with the decision criteria rather than a preference. Self-hosting pays when volume is high and steady enough to keep GPUs busy, when data must stay in a region, when you need adapters or models a provider does not offer, or when predictable cost matters. It loses when traffic is small or spiky, when no one can staff GPU on-call, or when the best model is only available hosted. Offer to settle it with a model: GPU-hours, expected utilization, engineering time, and the break-even volume, revisited when prices change. Then recommend something specific, often a hybrid: self-host the high-volume steady workloads and use a hosted provider for the long tail and for burst, with a gateway such as LiteLLM (LiteLLM) making the split invisible to teams.

Why not pick whichever engine benchmarks fastest?

Benchmarks answer a narrower question than the one being asked. The honest comparison covers your model, hardware and request mix, and includes operational cost: build steps, upgrade effort, observability, community and vendor support. TensorRT-LLM (TensorRT-LLM) may deliver more on NVIDIA hardware at the cost of an engine-build step; SGLang (SGLang) is worth testing for workloads with heavy shared prefixes. A defensible position is to standardise on vLLM for breadth and OpenAI-compatible simplicity, keep the interface behind the gateway so an engine can change, and run a recurring benchmark on your real traces to detect when another engine would pay for itself.

Leadership asks for a cost reduction of thirty percent

Start with measurement: what is utilization, what is the prefix-cache hit rate, what fraction of requests use the full context, what share of GPUs sit idle outside business hours. Then rank levers by risk. Low risk: right-size --max-model-len and scheduler limits to real traffic, fix cache-blind routing, consolidate per-team deployments into shared pools. Medium risk: LoRA multiplexing instead of per-customer deployments, scale-to-zero for non-interactive models. Higher risk, needing evaluation: FP8 or AWQ quantization and an FP8 KV cache. Say the quality cost of quantization out loud and propose an evaluation gate. Reject the tempting number, which is a bigger batch: it raises throughput and lowers the objective you are paid to meet.

We are two releases behind and a team wants the new feature. Do we upgrade?

Frame it as risk management over a two-week release cadence. Read the breaking changes for each skipped minor release, since removals happen on minor versions: for example, 0.29.0 made the V2 model runner the default and removed ten architectures, and 0.30.0 removed GPTQ activation-ordering support and made scale-out endpoints opt-in. Run the staged process: clean environment, evaluation set, benchmark, canary. Commit to a date rather than a yes or no, and propose a standing cadence so the question stops recurring. A pinned version that nobody maintains is a security and a hiring problem.

Why does the platform need a gateway when vLLM already has an API key?

Because the key authenticates some paths with one shared secret, offers no per-tenant identity, no quotas, no usage logs, no budget enforcement and no endpoint allowlist. The gateway is where identity, limits and accountability live, and where you can swap engines without touching tenants. Make it concrete with one sentence for a non-technical audience: the engine is a very fast worker with no front desk.

Short answers to have ready

• What does gpu_memory_utilization actually mean? The share of total GPU memory this instance may use for weights, activations and cache. Default 0.92 in 0.30.0.
• Why does vLLM have no swap space flag? V1 preempts by recomputation, not by swapping; --swap-space is gone.
• Prefill versus decode? Prefill is compute-bound and sets time to first token; decode is memory-bandwidth-bound and sets inter-token latency.
• What does chunked prefill do? Splits long prompts so decodes continue alongside them; on by default.
• Is prefix caching safe across tenants? Fast, but a timing side channel; use cache_salt per tenant.
• When do you use pipeline parallelism? Across nodes, when tensor parallelism would cross a slow link.
• What do you autoscale on? Waiting requests and KV usage, not CPU.
• Which flag replaced guided_json? The structured_outputs field with json, regex, choice or grammar.
• What replaced VLLM_ATTENTION_BACKEND? --attention-backend.
• What is wrong with latest? An untested release, with removals, arriving whenever the image is pulled.
• What is the first thing to read in a failed startup? The error above the Engine core initialization failed wrapper.
• Does vLLM work on Windows? Not natively; use WSL2.

Sixty-second self-test

Answer each aloud in one or two sentences.

1. What is the unit of failure for one vllm serve, and what does that imply for resilience?
2. Why do latency and preemptions rise while GPU utilization stays flat?
3. Name four things --api-key does not protect you from.
4. How do you size a long-context deployment without guessing?
5. Where does tensor parallelism stop being a good idea?
6. What does a per-tenant cache_salt defend against, and what does it cost?
7. Why is a startup probe different from a liveness probe for this workload?
8. Which three silent changes can alter output after an upgrade without any error?
9. When is another engine or a hosted API the better answer?
10. What would you refuse to build until you had measured the need?

If you stumble on more than two, reread the matching sections of the detailed guide before the interview.

Part three of three. At this level the problems are not failed commands. They are quiet: a replica that is fast in staging and slow in production, a cache that is warm for one tenant and poisoned for another, an upgrade that changes answers without an error. Start with the error table, then the platform habits, then the review checklists you can paste into a pull request template. Everything is checked against vLLM 0.30.0.

Production traps that appear at scale

Cumulative: everything from the earlier levels still applies. These cost money or trust rather than a failed start.

The trap behind most of these: reading utilization as health

GPU utilization is the metric everyone reaches for, and it is nearly useless for deciding whether an inference replica is healthy. Decode is memory-bandwidth-bound, so an engine can be at its limit with a utilization number that looks relaxed. The signals that predict user pain are queueing (vllm:num_requests_waiting, vllm:request_queue_time_seconds), cache pressure (vllm:kv_cache_usage_perc), preemption, and the latency percentiles themselves. Build the dashboard around those, and treat utilization as a cost metric, not a health metric.

The trap behind the rest: configuration that differs by hardware

The default max_num_seqs and max_num_batched_tokens depend on the GPU and on whether you are running the server or the offline interface. On a 160 GB class GPU both are generous; on an H100 class the server default for the token budget is 8192; on an A100, L4 or consumer card the server default drops to 2048 tokens and 256 sequences. A fleet with mixed pools therefore behaves differently per pool even though the manifests are identical. Set the values explicitly in the chart, per pool, and record them next to the measured capacity.

Traps in security and tenancy

These are the mistakes that become incident reports.

Dev mode in a shared cluster. Someone sets VLLM_SERVER_DEV_MODE=1 to use the sleep endpoint during a test and the manifest is copied. That flag exposes /collective_rpc, arbitrary RPC to the workers. Fail the deployment in an admission check if the variable is present.

Runtime LoRA loading left on. VLLM_ALLOW_RUNTIME_LORA_UPDATING plus reachable /v1/load_lora_adapter lets anyone who can call the API load files into the engine. Register adapters through a reviewed pipeline.

--trust-remote-code by habit. It runs Python from the model repository. Treat each use as a code-review item with a pinned --revision, and prefer models with native support after an upgrade.

A shared, writable compile cache. VLLM_CACHE_ROOT content is loaded and executed. Do not mount one cache volume across unrelated services, and do not make it world-writable to fix a permission error.

Metrics and gRPC exposed by accident. /metrics and --grpc are unauthenticated. Scrape from inside the cluster only.

SSRF through image URLs. A multimodal endpoint fetches whatever URL a caller sends, including internal addresses. Set --allowed-media-domains, keep VLLM_MEDIA_URL_ALLOW_REDIRECTS=0, and keep the download size and pixel limits at values you chose.

Client-supplied chat templates. --trust-request-chat-template is off by default for a reason. Keep it off for anything that is not fully trusted.

Open CORS. The default allowed origins is a wildcard. Tighten it, or terminate in a proxy that does.

The key on the command line. --api-key appears in the process list. Pass VLLM_API_KEY from a secret instead.

Platform-ownership habits

Habits compound; these are the ones that pay back most for someone who owns the tool for other teams. They are ordered by leverage, roughly.

1. Put a gateway in front, always. Identity, quotas, allowlists, usage logs and engine independence all live there. Everything else on this list is easier once it exists. LiteLLM (LiteLLM) is one common choice, and a plain proxy handles authentication and path allowlists.

2. Write down the capacity number. For each model, run vllm bench serve on staging hardware at rising concurrency and record the point where waiting requests grow and preemptions begin. Put the number, the date, the version and the hardware in the service catalogue. A capacity figure with no date goes stale at the next release.

3. Pin everything, and make upgrades boring. Exact version, image digest, model revision. A scheduled cadence, a checklist that includes reading the breaking changes, an evaluation set, a benchmark, and a canary. When upgrades are routine, nobody postpones them until three removals have stacked up.

4. Make the safe configuration the default. Ship a chart or base manifest where the safe flags are already set and the dangerous ones are absent, and enforce it with admission policy. Teams will use the paved road if it is faster than the alternative. Combine with Argo CD (Argo CD) so drift is reverted.

5. Alert on the cause, not the symptom. Queue depth, KV usage and preemption warn before latency objectives break. Pair them with a latency alert for the objective itself. Keep alert thresholds in version control and recalibrate them after each hardware or model change.

6. Persist and protect the startup path. A persisted, private VLLM_CACHE_ROOT, weights on local fast storage, a startup probe, and a drain period on shutdown. If restart time matters, test it as a feature: time a cold replica from scheduling to first successful request every release.

7. Own the evaluation set. A small standing set of prompts with expected behaviour, run on every engine, model or quantization change, catches the silent regressions that smoke tests miss. It does not have to be large; it has to be run.

8. Decide the sampling policy once. Either accept each model's generation_config.json defaults, or set --generation-config vllm and specify sampling in requests. Write the choice down in the tenant contract so a model swap does not change behaviour invisibly.

9. Keep the engine behind an interface you control. Tenants speak the OpenAI-compatible API to your gateway. Flag churn, an engine switch, or a move to another provider is then your internal change and not their migration.

10. Review deprecation warnings monthly. Engine logs name the removal version of anything deprecated. Collect them in staging and triage them as backlog. The 0.29 and 0.30 removals (the V1 model runner, the old entrypoints, several environment variables becoming flags) were all announced first.

11. Practise the failure. Kill a replica at peak in staging. Remove a GPU. Block the model store. Run the incident playbooks as drills, because a playbook never run is a hypothesis.

12. Say no in writing, with a reason and an alternative. A refusal of a dev-mode flag or an unpinned model is only durable if it points to the approved path.

Cost traps

A replica per team. The cheapest large saving is usually consolidating small deployments into shared pools with tenancy controls. The expensive mistake is doing it without the controls.

Generous --max-model-len for everyone. It reserves capacity and invites a few huge requests that crowd out many small ones. Tier contexts, and price the long tier.

Optimising throughput past the objective. A bigger token budget raises tokens per second and ruins time to first token. Report goodput under the objective.

Quantization without evaluation. FP8 or AWQ may be fine for your task; that is a measurement, not an assumption. Gate it.

Speculative decoding enabled globally. It can help latency at low concurrency and hurt throughput under load. Enable per model after a benchmark at your real concurrency.

Idle GPUs outside business hours. Scale-to-zero through KServe or KEDA fits internal and batch models, with cold starts of minutes accepted. It does not fit interactive products.

Autoscaling on CPU. The inference pod's CPU stays low when it is saturated. Scale on waiting requests and KV usage, and add a cooldown that respects the startup time, otherwise you pay for replicas that are still loading when the spike has gone.

Review checklist for a pull request

Paste into the pull-request template for any change touching a vLLM deployment.

• Image tag and digest are pinned; no latest.
• No VLLM_SERVER_DEV_MODE, no VLLM_ALLOW_RUNTIME_LORA_UPDATING, no --profiler-config, no --enable-tokenizer-info-endpoint.
• --trust-remote-code absent, or an exception is linked and --revision is pinned.
• --host, ports and network policy restrict access to the gateway; /metrics is not routed publicly.
• Secrets come from the secret manager; the key is passed as VLLM_API_KEY, not as an argument.
• --max-model-len, --max-num-seqs, --max-num-batched-tokens and --gpu-memory-utilization are explicit and match a recorded capacity measurement.
• Admission limits (--max-num-queued-reqs, --max-num-queued-tokens) are set for shared pools.
• Startup probe covers load plus compile time; readiness follows it; the Service is not named vllm.
• /dev/shm is mounted in memory (or --ipc=host) for any multi-GPU pod.
• Compile cache is persisted and private; weights come from the internal store.
• Every flag in the diff exists in the pinned version (checked with vllm serve --help=listgroup or the versioned docs), with none of the removed ones: --task, --swap-space, VLLM_USE_V1, VLLM_ATTENTION_BACKEND, guided_* fields.
• Evaluation and benchmark results for the change are attached.
• A rollback path and a drain period are described.

Review checklist for a design document

Use these questions in a design review before anything is built.

Requirements. What are the latency objectives for first token and inter-token latency, and who agreed them? What is the context length, and does the product need it? What is the traffic shape: steady, diurnal, bursty?

Capacity. What is the measured KV capacity per replica, and how was it measured? How many tenants share it, and what happens at the limit: queue, shed or fail?

Topology. Why this parallelism, and is there a smaller one that fits? If multi-node, how was the fabric verified? What is the failure domain of one instance?

Security. What is exposed, to whom, through what? Where is authentication enforced, and which paths are allowlisted? Is inter-node traffic isolated? What stops one tenant from reading another's cache timing?

Tenancy and cost. Which pool does each class of tenant use? How is usage attributed? What are the levers if cost must drop thirty percent, and what quality risk does each carry?

Operations. Which dashboards and alerts exist? What is the upgrade cadence and who owns it? What is the rollback? Which playbooks cover the likely failures?

Exit. What would make us switch engines or move to a hosted API, and is the interface designed so the switch is internal?

Practice cards

Quick drills to run with a colleague or alone. Cover the right-hand column.

Operational recipes

Recipes assume vLLM 0.30.0. Verify flags against your pinned version before use.

Measure capacity on a staging replica

vllm serve Qwen/Qwen3-0.6B --max-model-len 8192 --gpu-memory-utilization 0.9 &
vllm bench serve --model Qwen/Qwen3-0.6B \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --num-prompts 500
curl -s http://localhost:8000/metrics | grep -E "kv_cache_usage|num_requests_waiting|num_preemptions"

Repeat at rising concurrency and record the point where preemptions first appear. The benchmark flags above come from the command documentation; confirm any concurrency-related option with vllm bench serve --help on your version.

Persist the compile cache and run non-root in Docker

docker run --runtime nvidia --gpus all --ipc=host --user 2000:0 \
  -v vllm-cache:/home/vllm/.cache/vllm \
  -e VLLM_NO_USAGE_STATS=1 -e VLLM_API_KEY \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-openai:v0.30.0 --model Qwen/Qwen3-0.6B

The image has a built-in vllm user with UID 2000 and group 0, and non-root caches belong under /home/vllm/.cache. Binding the published port to 127.0.0.1 keeps it off the network until a gateway is in front.

Kubernetes container fragment with a startup probe

containers:
  - name: engine
    image: vllm/vllm-openai:v0.30.0
    args: ["--model", "/models/served", "--max-model-len", "16384"]
    env:
      - name: VLLM_API_KEY
        valueFrom: { secretKeyRef: { name: vllm-secrets, key: api-key } }
    resources:
      limits: { nvidia.com/gpu: 1 }
    startupProbe:
      httpGet: { path: /health, port: 8000 }
      periodSeconds: 10
      failureThreshold: 90
    readinessProbe:
      httpGet: { path: /health, port: 8000 }
      periodSeconds: 5
    volumeMounts:
      - { name: dshm, mountPath: /dev/shm }
volumes:
  - name: dshm
    emptyDir: { medium: Memory }

The generous startup threshold covers load and compile. The memory-backed /dev/shm is for multi-GPU communication, and it is harmless on one GPU. Model weights should come from an internal volume mounted at /models/served, and the Service for this Deployment must not be named vllm.

Isolate engine defaults from model defaults

vllm serve <model> --generation-config vllm

Run the same request with and without the flag. If outputs differ, the model's generation_config.json is influencing sampling. Decide the policy and document it.

Triage a failed start

VLLM_LOGGING_LEVEL=DEBUG vllm serve <model> 2>&1 | tee start.log
vllm collect-env
nvidia-smi

Read upward from Engine core initialization failed to the real error. If it is a CUDA graph crash, confirm with --enforce-eager, and include vllm collect-env output in any bug report.

Verify the multi-node transport

Run the sanity script from the troubleshooting page with NCCL_DEBUG=TRACE and torchrun --nproc-per-node=<GPUs>. A line containing via NET/IB/GDRDMA means RDMA is in use; NET/Socket means plain TCP and a slow tensor-parallel path. If the interfaces are wrong, set VLLM_HOST_IP, NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME.

A final list for the first week of ownership

If you inherit a vLLM deployment on Monday, do these in order. First, find out what is reachable and from where, and close it with a gateway and a network policy. Second, record the version, image, model revisions and every non-default flag, and pin them. Third, build one dashboard with queue depth, KV usage, preemption, and the latency percentiles, and one alert on each of the first three. Fourth, measure capacity on staging and write the number down. Fifth, scan the manifests for dev mode, runtime LoRA updating and --trust-remote-code. Sixth, write the upgrade checklist and a first draft of the playbooks. Seventh, schedule the first upgrade rehearsal. None of it needs a new tool, and all of it prevents the incidents that this series has described.