The Complete Ray Serve Guide

This is part one of three. It covers everything you need to do real work with Ray Serve, not a teaser. By the end you can turn a Python function or a machine-learning model into a web API, run several copies of it, make one piece of your service call another, describe the whole thing in a configuration file, and read the status output when something goes wrong. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas in Serve only stick once you have watched your own service start, answer a request, and survive a crash.

You do not need to know Ray beforehand. You do need to be comfortable with basic Python (classes, functions, pip) and to have seen an HTTP request, for example with curl. If you have never built a web API at all, skim the FastAPI guide first; Serve borrows from it heavily.

What Ray Serve is, and the problem it solves

A trained model on your laptop is not a product. Somebody has to be able to send it an input over the network and get a prediction back, many times a second, without the service falling over. That job is called online inference, and Ray Serve is a library for doing it.

Serve is a scalable model-serving library built on top of Ray. It is framework-agnostic: PyTorch, TensorFlow, Keras, scikit-learn, or any ordinary Python code can sit behind it. The documentation describes it as built for end-to-end inference APIs that mix several models with business logic, rather than only "tensor in, tensor out". That phrase matters. Real prediction services rarely consist of a single model call. They clean the input, look something up, call a model, perhaps call a second model with the first one's answer, and then shape a response. Serve lets all of that live in one Python program that you can scale as a unit.

The diagram is the whole idea. You write a normal class, mark it as a deployment, and Serve runs as many copies of it as you ask for, spreads incoming requests across them, and exposes a single address to callers.

Two consequences of that design explain most of what follows.

Your service is ordinary Python. There is no special model format and no separate serving language. If you can call your model from a script, you can serve it. This is why people who have used heavier model servers often find Serve refreshingly small: the unit of work is a class you already know how to write.

Scaling is a setting, not a rewrite. The same class that runs as one copy on your laptop runs as ten copies on a cluster, because the copies are managed by Ray. You change a number, or you let Serve change it for you based on load. We will do the first version of that in this guide and leave the sophisticated version for later.

What people use it for:

What came before Serve

To see why Serve exists, look at the options people used before it, and at where each one hurts.

The simplest option is a web framework around your model: a Flask or FastAPI app that loads the model at start-up and calls it inside a route function. This is a perfectly good way to start, and for one small model on one machine it is often all you need. The trouble appears when traffic grows. To use more CPU cores you start more worker processes, and each loads its own copy of the model. To use more machines you add a load balancer and a deployment system. If one step of your pipeline is slow and the others are fast, you cannot scale that step alone, because the whole app is one blob. If the model needs a GPU, you now have to decide by hand how many workers share it.

The second option is a dedicated model server that expects your model in a particular format and exposes a fixed interface. These are strong when your model fits the format and the request is just a tensor. They are less comfortable when your service needs arbitrary Python between the request and the model, such as calling a feature store, a vector database, or another model chosen at run time.

The third option is hand-built infrastructure: containers on Kubernetes, a queue, a custom autoscaler. This works and gives total control, but it means your team now owns a serving platform instead of a model.

Serve sits between these. You keep writing Python, which is the comfort of the first option. But replicas, load balancing, scaling and composition are handled by Ray, which removes most of the pain of the third. The honest framing, which the rest of this course will repeat, is that Serve pays off when you need several replicas, shared GPUs, multi-step composition, or LLM serving. If you have one small model on one machine, a plain FastAPI app may be the better choice, and Serve can wait until you outgrow it. The FastAPI guide covers that starting point, and you will see later that Serve can wrap a FastAPI app rather than replace it.

The mental model: four nouns

Serve has a small vocabulary. Learn these four words and the rest of the documentation becomes readable.

A deployment is a Python class or function decorated with @serve.deployment. It is the unit of scaling: the thing you ask Serve to run several copies of. A deployment is a description, not a running process.

A replica is one running copy of a deployment. Technically it is a Ray actor, which is a long-lived Python process managed by Ray that holds state (for example, a loaded model) between requests. If you ask for three replicas, there are three such processes, each with its own copy of whatever your class loaded in its constructor.

An application is one or more deployments bound together. You create the binding by calling .bind() on a deployment, which returns an application object without running anything. The top-most deployment in an application is called the ingress: it is the one that receives outside traffic. An application has a name and a route_prefix, the URL path under which it is reachable.

A handle, called a DeploymentHandle, is how one deployment calls another at run time. When your ingress needs the preprocessing deployment, it uses a handle. Handles are what make composition possible, and we meet them in their own section.

Behind the scenes there are two more actors that you do not write but should know exist. The controller is a single, long-lived actor that records which deployments should exist, starts and stops replicas, and drives autoscaling. The proxy is a per-node actor that accepts incoming HTTP traffic and routes each request to a replica. You will see both named in logs and status output, so it helps to recognise them early.

One vocabulary note. Ray itself is a general framework for running Python in parallel across machines, and Serve is only one library built on it. You do not need to learn Ray tasks or actors to use Serve at this level. If you are curious about the foundation, the Ray guide in this catalogue covers it.

Installing Ray Serve and checking the setup

Serve ships inside the ray Python package and is versioned together with Ray. There is no separate Serve version number. This guide was written against the Ray 2.58 line, and the documentation's current stable release at the time of writing is Ray 2.58.x. Because versions move quickly, check what pip will give you before you start:

pip index versions ray

Ray supports Linux on x86_64 and aarch64, macOS on Apple Silicon, and Windows in beta. Multi-node clusters on Windows are experimental, so if you are on Windows and want the smoothest path, use WSL2. The supported Python versions are 3.10, 3.11 and 3.12, with 3.13 in beta. Older Python versions are not supported by current Ray, so if python --version prints 3.9 or lower, install a newer interpreter first.

Always use a virtual environment. Ray pulls in many dependencies, and you do not want them tangled with the rest of your machine.

python -m venv .venv
source .venv/bin/activate
pip install -U "ray[serve]"

On Windows PowerShell the equivalent is:

py -m venv .venv
.venv\Scripts\Activate.ps1
pip install -U "ray[serve]"

The quotes around ray[serve] are not decoration. In zsh, the default shell on macOS, square brackets are special characters, and an unquoted ray[serve] fails with a "no matches found" message. Quoting it works everywhere.

The serve in square brackets is an extra: it asks pip to install Ray plus the additional packages Serve needs. There are other extras you may meet. ray[default] gives you the common set including the dashboard. ray[data,train,tune,serve] installs the machine-learning libraries together. ray[llm] adds the LLM-serving stack with vLLM and normally wants a GPU. For this guide, ray[serve] is all you need.

Verify the installation with a command that imports the library and prints its version, then check that the command-line tool is on your path:

python -c "import ray; from ray import serve; print(ray.__version__)"
serve --help

The first command should print a version such as 2.58.0. The second should list subcommands including run, build, deploy, status, config and shutdown. If the first fails with ModuleNotFoundError: No module named 'ray', your virtual environment is not active. If the second says command not found, the same cause is likely: the serve program lives in the environment's bin folder.

If you plan to run Serve inside a container, remember that Ray wants roughly 30 percent of the machine's memory as shared object store memory, and Docker's default shared-memory allowance is small. The Ray documentation recommends raising it with the --shm-size flag when you start the container. The Docker guide explains the container side of that.

Your first deployment, step by step

Create a file called hello.py. We build it in stages so you can see what each line is for.

from ray import serve
from starlette.requests import Request


@serve.deployment
class Hello:
    def __call__(self, request: Request) -> str:
        return "hello"


app = Hello.bind()

Read it from the top. The first line imports the serve module from Ray. The second imports Request, a class from Starlette, the small web toolkit that Serve uses to represent incoming HTTP requests. You do not install Starlette separately; it comes along with Serve.

The decorator @serve.deployment turns the class below it into a deployment. Nothing runs yet. The class has one method, __call__. In Python, defining __call__ makes an instance callable like a function, and Serve uses it as the entry point: when an HTTP request arrives, Serve calls the instance with a Request object and sends back whatever you return. Here we return the string "hello".

The last line, app = Hello.bind(), builds the application. bind does not start anything. It records that you want the Hello deployment, with whatever constructor arguments you pass (none here), as an application called app. That name matters, because the command you are about to run refers to it.

Now run it from the same folder:

serve run hello:app

The argument hello:app is an import path: the module name (the file hello.py without the extension), a colon, and the variable inside it. Serve imports hello, takes the object named app, starts a local Ray instance if one is not already running, and deploys the application. The command blocks and keeps running. By default it serves on http://127.0.0.1:8000. Leave it running and open a second terminal:

curl http://127.0.0.1:8000/

You should see hello. Back in the first terminal you will see log lines for the request. Press Ctrl+C there to stop the service.

Congratulations: you have run an online inference service. It does not contain a model yet, but the machinery is complete. A request went through the proxy to a replica of your deployment and a response came back.

You can also run an application from inside Python rather than from the command line:

from ray import serve

handle = serve.run(app, name="hello", route_prefix="/")

serve.run deploys the application, gives it a name and a route prefix, and returns a handle to the ingress deployment. It is handy in notebooks and tests. For anything you keep, the command line and, later, the configuration file are the better habit.

Handling requests and returning JSON

A real service accepts input. In the first example the method received a Request and ignored it. Now we use it.

The Request object is a Starlette request, so it offers the usual accessors. request.query_params holds the values after the question mark in the URL. await request.json() reads a JSON body. Because reading the body is asynchronous, a method that does so must be declared async def.

from ray import serve
from starlette.requests import Request


@serve.deployment
class Greeter:
    def __init__(self, greeting: str = "Hello"):
        self.greeting = greeting

    async def __call__(self, request: Request) -> dict:
        body = await request.json()
        name = body.get("name", "stranger")
        return {"message": f"{self.greeting}, {name}!"}


app = Greeter.bind("Salam")

Notice three things. First, the constructor __init__ runs once per replica, when the replica starts. This is the right place to load a model, open a connection, or read a file, because you pay the cost once and every later request reuses the result. Second, the constructor takes an argument, and Greeter.bind("Salam") supplies it. Whatever you pass to bind goes to the constructor. Third, the method returns a dictionary, and Serve converts it to a JSON response.

Run it with serve run greeter:app and call it:

curl -X POST http://127.0.0.1:8000/ \
  -H "Content-Type: application/json" \
  -d '{"name": "Aya"}'

The reply is {"message": "Salam, Aya!"}.

Why do we care about the difference between __init__ and __call__? Because it is the single most important performance habit in model serving. Loading a model can take seconds or minutes. If you load it inside __call__, every request pays that cost. If you load it in __init__, only the start-up pays it. A beginner who puts pipeline("summarization") inside the request method will see a service that works but takes many seconds per call, and will blame the model. The model is fine; the loading is in the wrong place.

The other thing to understand is sync versus async. A method declared def runs in a thread pool on the replica, so slow blocking work in it does not freeze the replica. A method declared async def runs on the replica's event loop, which is faster for waiting on network calls but has a rule: never put a long blocking call (a heavy model forward pass, time.sleep) directly in it, because that stalls everything else the replica is doing. For a beginner, the safe rule is this: use async def when you need to await something such as request.json(), and keep the heavy computation short or move it into a plain def method that you call.

Serving a real model with FastAPI

Hand-parsing JSON gets tedious fast. Serve integrates with FastAPI, which gives you routing, input validation and automatic documentation, and the integration is one decorator. If you know FastAPI, you already know most of this section; the FastAPI guide is the place to learn it properly.

The pattern is to create a FastAPI app object, decorate your deployment class with @serve.ingress(api), and declare routes on the class with the app's decorators. The class becomes the ASGI application (ASGI is the interface standard that Python async web apps share).

from fastapi import FastAPI
from ray import serve

api = FastAPI()


@serve.deployment
@serve.ingress(api)
class Summarizer:
    def __init__(self):
        from transformers import pipeline

        self.pipe = pipeline("summarization")

    @api.post("/")
    def summarize(self, text: str) -> str:
        return self.pipe(text)[0]["summary_text"]


app = Summarizer.bind()

To run this you also need the libraries it imports:

pip install transformers torch
serve run text_ml:app

Look at the order of decorators. @serve.deployment is on the outside and @serve.ingress(api) sits just above the class. The route is declared with @api.post("/"), exactly as in a normal FastAPI app. The text: str parameter is read by FastAPI from the request, and the return annotation tells FastAPI how to shape the response.

The import of transformers is inside __init__ instead of at the top of the file. This is a deliberate choice that the Serve documentation recommends: a delayed import means the heavy library is imported on the replica, where the model runs, rather than in the process that merely builds the application object. It also lets different deployments in one application use different dependency versions later on.

Test it. Because text is a plain str parameter on a POST route, FastAPI expects it as a query parameter:

curl -X POST "http://127.0.0.1:8000/?text=Ray%20Serve%20lets%20you%20turn%20Python%20classes%20into%20scalable%20web%20services."

The first run downloads the default summarization model, which can take a few minutes and needs internet access, so be patient and watch the logs. After that, requests are fast. Since FastAPI is in control, you also get interactive documentation for free: open http://127.0.0.1:8000/docs in a browser to see and try your routes.

Which style should you use, the raw Request or FastAPI? For anything beyond a quick experiment, FastAPI. You get validation (a request with a missing field is rejected with a clear error before your code runs), documentation, and a style that other engineers recognise. The raw Request approach is fine for tiny services or when you need full control of the body.

Replicas and resources: running more than one copy

So far each deployment has had a single replica. The reason to use Serve at all is that you can ask for more, and the way to do it is an argument to the decorator.

from ray import serve
from starlette.requests import Request


@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 1},
)
class Model:
    def __init__(self):
        import os

        self.pid = os.getpid()

    async def __call__(self, request: Request) -> dict:
        return {"served_by_process": self.pid}


app = Model.bind()

num_replicas=2 tells Serve to run two copies. ray_actor_options describes what each replica needs from the machine. Here every replica reserves one CPU. Ray treats these numbers as logical resources: they are a promise used for scheduling, not a hard limit enforced on the process. If a machine has eight CPUs and each replica asks for one, Ray will place up to eight such replicas there. If you ask for more than the cluster has, the extra replicas wait, and the status output will tell you they are waiting for resources.

Run it and call it several times:

serve run scaled:app
# in another terminal
for i in 1 2 3 4 5 6; do curl -s http://127.0.0.1:8000/; echo; done

You will see two different process ids alternating or mixing in the answers. That is load balancing, made visible: the proxy spreads the requests across both replicas. Each replica ran its own __init__, so each has its own pid, and, with a real model, its own copy of the model weights in memory. This is the first thing to budget for. Two replicas of a 2 GB model need about 4 GB of memory, not 2.

For models that need a GPU, you ask in the same place:

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_cpus": 1, "num_gpus": 0.5},
)
class GpuModel:
    ...

Fractional GPUs are allowed. Two replicas that each request half a GPU share one physical card, which can save a lot of money when a model is small enough that a whole GPU would sit mostly idle. Note the word "logical" again: Ray will not stop one replica from using more than half of the card's memory. Fractions are a scheduling agreement, and keeping each model inside its share is your responsibility.

There is a second knob you will see in the documentation and should recognise: max_ongoing_requests. This limits how many requests a single replica handles at once. Its default is 5 in current versions (it was 100 before Ray 2.32, and was named max_concurrent_queries, a name you may still see in older tutorials). If every replica is already handling its maximum, new requests wait in a queue. For a model that processes one request at a time, a small number is sensible, and it is a safety valve that stops one replica from accepting more work than it can handle.

A deployment also has a name. By default it is the class name. You can set it explicitly with @serve.deployment(name="summarizer"), which makes status output easier to read when you have several deployments.

Calling one deployment from another

Real services have parts. Suppose a request carries messy text, and the model wants it cleaned first. You could do both in one class, but then cleaning and modelling scale together even if one of them is trivial. Serve lets you split them into two deployments and connect them.

The connector is the handle. When you pass one bound deployment into another's bind call, Serve gives the receiving class a DeploymentHandle for it in its constructor. You call the handle's method with .remote(...), and you get back a DeploymentResponse, which you can await.

from ray import serve
from starlette.requests import Request


@serve.deployment
class Preprocessor:
    def __call__(self, text: str) -> str:
        return text.strip().lower()


@serve.deployment
class Model:
    def __init__(self, pre):
        self.pre = pre

    async def __call__(self, request: Request) -> dict:
        text = (await request.json())["text"]
        clean = await self.pre.remote(text)
        return {"out": clean}


app = Model.bind(Preprocessor.bind())

Read the last line carefully, because it is where composition happens. Preprocessor.bind() creates a bound preprocessor. Model.bind(...) receives it as an argument, so Model's constructor gets pre, which is a handle to the preprocessor, not the preprocessor itself. In __call__, self.pre.remote(text) sends the text to a preprocessor replica and returns immediately with a response object. Awaiting it produces the cleaned string.

The top of the chain, Model, is the ingress. It is the only deployment reachable from outside. Preprocessor has no route of its own; it is an internal step. This is a useful property: you can scale the two parts independently, give them different resources, and give the preprocessor no GPU while the model has one.

@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 0.5})
class Preprocessor:
    ...


@serve.deployment(num_replicas=3, ray_actor_options={"num_cpus": 1, "num_gpus": 1})
class Model:
    ...

Why await? The DeploymentResponse is a future: a placeholder for a result that is still being computed on another replica. Awaiting it lets the replica serve other requests while it waits. From ordinary synchronous code, you can call .result() on the response instead, which blocks until the answer is ready. Inside an async def method, always prefer await.

If you have read older tutorials, you may have seen something called a deployment graph with a DAGDriver, and handle types named RayServeHandle. Those belong to older Ray versions and are replaced by the .bind() composition and DeploymentHandle shown here. If you meet them, translate rather than copy.

Composition is also where cross-links to the rest of this catalogue become natural. A model-serving pipeline often reads features from a store, and logs its predictions for monitoring. When you reach for those, look at the Feast guide for features and the Evidently guide for monitoring.

The command line and the config file

Running serve run in a terminal is perfect for development. It is the wrong tool for something that must keep running. For that, Serve has a declarative workflow: describe what you want in a YAML file, and tell a running Ray cluster to make it so.

The command-line tool has a handful of subcommands, and it is worth knowing each by its purpose:

The full workflow on one machine is as follows. First, generate a config from your import path:

serve build text_ml:app -o serve_config.yaml

This writes a YAML file that describes your application. A trimmed example looks like this:

proxy_location: EveryNode
http_options:
  host: 0.0.0.0
  port: 8000
applications:
  - name: text_app
    route_prefix: /
    import_path: text_ml:app
    runtime_env:
      pip: ["transformers", "torch"]
    deployments:
      - name: Summarizer
        num_replicas: 2
        max_ongoing_requests: 10
        ray_actor_options:
          num_cpus: 1

Read it as a description of the same things you set in Python. applications is a list, so one cluster can host several applications, each with its own name and route_prefix. import_path is the same module:variable string you passed to serve run. runtime_env says what packages the replicas need. deployments lets you override per-deployment settings such as the number of replicas, so you can change scaling without touching the Python code.

The point of the file is that it can be committed to Git, reviewed, and applied repeatedly. The Python code defines what the service does; the YAML defines how it runs. That separation is the reason most teams use the config file in production.

Second, start a Ray cluster. On a single machine, one head node is enough:

ray start --head

Third, deploy the config and check the result:

serve deploy serve_config.yaml
serve status

Here is something that surprises newcomers. When serve deploy prints a success message, it means only that the cluster received the request. The deployment carries on in the background: replicas must start, models must load. So after deploying, run serve status repeatedly until the application reports that it is running. A model that takes two minutes to load will show a deploying state for two minutes, and that is normal.

To talk to a cluster that is not on your machine, the deploy command takes an address for the Ray dashboard, which listens on port 8265:

serve deploy serve_config.yaml -a http://<head-node-address>:8265

When you are done, clear the cluster:

serve shutdown
ray stop

The import path deserves a warning, because it is where many first deployments fail. The documentation states that the import path (for example text_ml:app) must be importable by Serve at runtime. On your laptop, the folder you run from is on Python's search path, so it works. On a cluster, the replicas are different processes, possibly on different machines, and they do not automatically have your file. The fix is to ship your code with the application: through a working_dir in the runtime_env, through a container image that contains it, or through a remote location the cluster can download from. For learning on one machine you will not hit this; remember it for the day you move to a real cluster.

Scaling with load, and a first look at batching

Setting num_replicas to a fixed number is fine until traffic changes. At three in the morning you pay for replicas that do nothing, and at lunchtime the same number may be too few. Autoscaling lets Serve adjust the replica count to the load.

You enable it by giving the deployment an autoscaling_config instead of a fixed num_replicas:

@serve.deployment(
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 5,
        "target_ongoing_requests": 2,
    },
    max_ongoing_requests=5,
)
class Model:
    ...

Here is the idea, in plain words. Serve measures how many requests each replica is working on at once, on average. You tell it the target you are comfortable with, target_ongoing_requests. If the average is above the target, Serve adds replicas, up to max_replicas. If it is well below, Serve removes replicas, down to min_replicas. The upper and lower bounds are your guard rails: max_replicas stops a traffic spike from creating unlimited cost, and min_replicas keeps some capacity warm.

A few details prevent confusion. The default min_replicas is 1, and the default max_replicas is also 1, which means that if you add an autoscaling config but forget max_replicas, nothing will ever scale up. The default target_ongoing_requests is 2. Scaling up waits for a delay (30 seconds by default) before acting, and scaling down waits much longer (600 seconds by default), so a short quiet moment does not tear down capacity you will need a minute later. That slow scale-down is deliberate, not a bug. Finally, do not set both num_replicas and autoscaling_config on the same deployment; choose one.

You can set min_replicas to 0 to save money when the service is idle, but then the first request after a quiet period must wait for a replica to start and load its model. For a model that loads in seconds that is acceptable; for one that loads in minutes it will feel like an outage. Start with a minimum of 1 and reduce it only when you have measured the cold start.

Autoscaling changes the number of replicas. Batching changes how much work each replica does per call, and for neural network models it is often the bigger win. GPUs and vectorised libraries process a group of inputs together almost as cheaply as one. Serve offers @serve.batch to collect individual requests into a group automatically.

from ray import serve
from starlette.requests import Request


@serve.deployment
class Batched:
    @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1)
    async def handle(self, items: list[int]) -> list[int]:
        return [i * 2 for i in items]

    async def __call__(self, request: Request) -> int:
        return await self.handle(int(request.query_params["n"]))


app = Batched.bind()

Notice the contract. The decorated method is async, it receives a list of inputs, and it must return a list of the same length, in the same order. The caller, however, passes one item and gets one result: __call__ sends a single integer and receives a single integer. Serve does the grouping behind the scenes. It waits up to batch_wait_timeout_s seconds (0.1 here), or until max_batch_size items have arrived (8 here), whichever comes first, then runs the method once on the whole group and hands each caller its own result.

The trade-off is latency against throughput. A longer wait gives bigger batches and better hardware use, but each request may wait a little longer. The documentation advises choosing a wait time below your latency budget minus the inference time, and using powers of two for the batch size. The defaults are a batch size of 10 and a wait of 0.01 seconds. You do not need to batch a service until you measure that you need it, but knowing the tool exists will save you from inventing your own.

Configuration you will meet early

Beyond replicas and resources, a few settings come up in nearly every real project. You do not need to memorise them. You need to know where to look.

Per-deployment options are set either in the decorator or, for an existing application, in the config file's deployments list. The decorator is the default; the YAML overrides it. If a value in the file disagrees with the code, the file wins, which is intentional: it lets an operator change scaling without a code change.

Runtime environments (runtime_env) describe the Python packages and files a replica needs. In the config file you saw pip: ["transformers", "torch"]. You can also set one per deployment with ray_actor_options={"runtime_env": {"pip": [...]}}, which lets two deployments in one application use different versions of a library. A caution from the documentation: avoid packages that compile from source at start-up, because that burns resources and can destabilise a cluster. For production, prefer to bake dependencies into a container image, as described in the Docker guide.

Logging is controlled with a LoggingConfig per deployment, available from Ray 2.9. It sets the log level, a JSON output format and whether to log each request. Logs live on the node under /tmp/ray/session_latest/logs/serve/, and the Python logger is called ray.serve. When a deployment fails to start, the answer is almost always in the replica logs there.

The proxy can run on every node or only on the head node, controlled by proxy_location; the config file generated by serve build shows EveryNode. On one laptop this changes nothing.

Environment variables matter occasionally. RAY_ADDRESS and RAY_DASHBOARD_ADDRESS tell the command-line tools which cluster to talk to, so you do not have to pass -a every time.

Authentication is not something Serve does for you. The HTTP endpoints have no built-in login. If your service will be reachable by anyone other than you, put an API gateway or ingress with authentication in front, or add a check inside your FastAPI app. Secrets such as API keys should come from the environment of the replica or from a secret manager and be read in __init__; never write them into the config file you commit. The Kubernetes guide explains the secret mechanisms used on the cluster side.

Common errors and how to read them

Most Serve problems fall into a few families. The skill is to look in the right place first. Two places answer almost everything: serve status, and the replica logs in /tmp/ray/session_latest/logs/serve/.

The status stays stuck in a deploying or unhealthy state. This is the most common complaint. It means the replicas could not start. The reasons are usually one of three. The constructor raised an exception, for example a missing file or a failed model download. The cluster does not have the resources you asked for, so the replica is waiting; the status will say it is waiting for resources, and the fix is to ask for less or add capacity. Or the options passed in ray_actor_options are wrong. Open the replica log: the Python traceback is there, and it names your file and line.

ModuleNotFoundError for your own module, or a failure to import the application. The import path is not importable where the replica runs. On one machine, check that you run serve run or serve build from the folder containing the file, and that the module name matches the file name without .py. On a cluster, ship the code with working_dir or a container image, as described earlier.

Cannot connect to Ray, or a failure to reach the head node. You are running a command that needs a cluster, and none is running at the address. Start one with ray start --head, or correct RAY_ADDRESS. Note that serve run starts a local instance for you, but serve deploy and serve status talk to an existing one.

Connection refused on port 8265. The dashboard is not reachable from where you are. On the same machine, check Ray is running. From another machine, you will need the head node's address and a network path to it, and you should think carefully about the security warning above before opening one.

A replica dies with an actor-died error. The process was killed, very often by the operating system for using too much memory. Two replicas of a large model use twice the memory of one. Reduce the replica count or batch size, or request more memory for the replica through ray_actor_options.

Requests are slow or time out. First ask whether the model is being loaded inside __call__ instead of __init__. Then ask whether a blocking call sits inside an async def. Then ask whether max_ongoing_requests is so small, relative to traffic, that requests are queuing. Only then look at the model itself.

Complaints about both num_replicas and autoscaling_config. You set both on one deployment. Remove one.

Validation or import errors after upgrading Ray. Recent Ray releases expect Pydantic version 2. Pydantic 1 has been deprecated, and the installation documentation announced that support would be dropped in release 2.56. If a library in your environment pins the old version, you will see import errors; upgrade that library or use a clean environment.

A certificate error during model or package download. If you are behind a gateway that inspects HTTPS, make Python trust the organisation's certificate as described in the installation section, rather than disabling verification.

Putting it all together

Here is one small end-to-end project that uses most of the guide. We will build a sentiment service with two deployments, validate its input with FastAPI, scale the model, write a config, deploy it to a local Ray cluster, and call it. The model is a small scikit-learn text classifier trained inside the file, so you need no download beyond the library itself.

First install the extras:

pip install scikit-learn

Then create sentiment.py:

from fastapi import FastAPI
from pydantic import BaseModel
from ray import serve

api = FastAPI()


class Review(BaseModel):
    text: str


@serve.deployment
class Cleaner:
    def __call__(self, text: str) -> str:
        return " ".join(text.lower().split())


@serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 1})
class Classifier:
    def __init__(self):
        from sklearn.feature_extraction.text import TfidfVectorizer
        from sklearn.linear_model import LogisticRegression
        from sklearn.pipeline import make_pipeline

        texts = [
            "great product, works well",
            "excellent quality and fast delivery",
            "love it, very happy",
            "terrible, it broke in a day",
            "awful experience, want a refund",
            "bad quality and slow support",
        ]
        labels = ["positive", "positive", "positive", "negative", "negative", "negative"]
        self.model = make_pipeline(TfidfVectorizer(), LogisticRegression())
        self.model.fit(texts, labels)

    def predict(self, text: str) -> str:
        return self.model.predict([text])[0]


@serve.deployment
@serve.ingress(api)
class Api:
    def __init__(self, cleaner, classifier):
        self.cleaner = cleaner
        self.classifier = classifier

    @api.post("/sentiment")
    async def sentiment(self, review: Review) -> dict:
        clean = await self.cleaner.remote(review.text)
        label = await self.classifier.predict.remote(clean)
        return {"text": clean, "label": label}


app = Api.bind(Cleaner.bind(), Classifier.bind())

Take it apart. Review is a Pydantic model: FastAPI uses it to validate the JSON body, so a request without text is rejected before your code runs. Cleaner is a trivial preprocessing step. Classifier trains a tiny model in __init__, once per replica, and exposes a predict method. Because it has two replicas, two copies of the model exist. Api is the ingress: it declares the route with FastAPI, receives handles to the other two in its constructor, and chains the calls. The last line builds the application by binding the three deployments together.

One new detail: self.classifier.predict.remote(clean) calls the predict method by name through the handle. When a deployment has a method other than __call__ that you want to reach, you call it as an attribute of the handle followed by .remote.

Run it first the quick way, to check that it works:

serve run sentiment:app

curl -X POST http://127.0.0.1:8000/sentiment \
  -H "Content-Type: application/json" \
  -d '{"text": "  Great PRODUCT, very happy  "}'

The reply is a JSON object with the cleaned text and a label, which should be positive for this input. Send an obviously negative sentence to see the other label. Send {} and read the 422 validation error from FastAPI. Stop the service with Ctrl+C.

Now do it the production way. Build the config, start a cluster, deploy and watch:

serve build sentiment:app -o sentiment_config.yaml
ray start --head
serve deploy sentiment_config.yaml
serve status

Edit sentiment_config.yaml to raise the classifier to three replicas, redeploy, and watch serve status pick up the change without you touching the Python file. Call the service again to confirm it still answers. When you are finished:

serve shutdown
ray stop

Finally, store the work properly. Put sentiment.py, the YAML and a requirements.txt in a Git repository. That repository is a complete, reviewable description of a service. When you are ready to run it for real users, the next step is a container image and a Kubernetes manifest, and the Docker guide and Kubernetes guide are the natural continuation.

What you can now do, and what comes next

You can now take a Python class and turn it into a service: decorate it, bind it, run it with serve run, and call it with curl. You understand the four nouns, deployment, replica, application and handle, and you can explain why a model belongs in __init__. You can run several replicas, give them CPUs or a fraction of a GPU, wire two deployments together, describe the whole application in a YAML file, deploy it to a local Ray cluster, and read serve status and the logs when something misbehaves. You know the most common errors by family, and you know to check the date on a tutorial because several option names changed in 2024.

That is enough to ship a first internal service. If you work for an employer in the Gulf or in Egypt, one practical note applies before you deploy anything with real user data: decide where the cluster runs. Data-residency rules at many organisations require that personal data stay inside the country or region, and your replicas, logs and any model downloads all count as places where data may travel. Pick your cloud region on purpose.

The mid-level guide goes under the hood: how the controller, proxy and replicas cooperate, how requests are routed and queued, deeper autoscaling settings, model multiplexing, streaming responses, testing a deployment, and running Serve on Kubernetes with KubeRay. The senior guide covers failure modes, security, multi-tenancy, cost and upgrades, and when to choose something else.

Where to go next in this catalogue depends on what you are serving. For language models, the vLLM guide covers the engine that Ray Serve LLM can drive. For comparison with other serving tools, see BentoML, KServe and Triton. For metrics and dashboards on a running service, see Prometheus and Grafana.

Sources

• Ray Serve documentation
• Ray Serve API reference
• Ray installation guide
• Serve production guide
• Handling dependencies
• Advanced autoscaling
• Dynamic request batching
• Monitoring Ray Serve
• Deploying on VMs
• Ray Serve LLM
• Ray releases on GitHub

This file prepares you to talk about Ray Serve in an interview for a junior or entry-level machine-learning engineering role. Interviewers at this level are not looking for cluster-operations war stories. They want to know that you understand what the tool is for, that you can describe its few core ideas in plain words, and that you have actually run something. Read the model first, then the commands, then the traps, then practise the answers aloud.

The thirty-second answer

Ray Serve is a Python library for serving models and inference logic as scalable online APIs. You write an ordinary class, decorate it with @serve.deployment, and Serve runs several copies of it as replicas on Ray, load balances requests across them, and lets you scale them by changing a number or by enabling autoscaling.

It is framework-agnostic, so the model can be PyTorch, scikit-learn or plain Python, and it is built for services made of several steps rather than a single model call. I would use it when I need multiple replicas, shared GPUs, a pipeline of models, or LLM serving, and I would use a plain FastAPI app when one small model on one machine is enough.

If the interviewer gives you only one minute, say that, then add one sentence from your own experience, such as "I built a two-deployment sentiment service with a cleaner and a classifier and deployed it from a YAML config".

The model you must be able to draw

Be ready to sketch this on a whiteboard in under a minute.

Four nouns carry the whole subject, and interviewers listen for them.

A deployment is a Python class or function decorated with @serve.deployment. It is the unit of scaling and it is a description, not a running process. A replica is one running copy of it, implemented as a Ray actor, a long-lived Python process that holds state such as a loaded model. An application is one or more deployments tied together with .bind(); the top one is the ingress, the only one that receives outside traffic, and the application has a name and a route prefix. A deployment handle is the object one deployment uses to call another; handle.method.remote(x) returns a response you can await.

Two background components get named less often but are worth a sentence. The controller is a single actor that tracks what should exist and does the scaling. The proxy is a per-node actor that accepts HTTP traffic. If you can say "the proxy receives the request and sends it to one of the ingress replicas, and the controller keeps the right number of replicas alive", you have shown the shape of the system.

Where does Ray fit? Ray is the general framework for running Python across machines; Serve is one library on top of it. A beginner does not need to program Ray directly, but should know that replicas are Ray actors and that the "resources" in ray_actor_options are Ray's scheduling units.

What happens when you call .bind() and serve run

This is a favourite follow-up, because it separates people who have typed the code from people who have understood it.

Hello.bind() returns an application object and starts nothing. It records which deployment to run and with which constructor arguments. serve run hello:app takes an import path: the module, a colon and the variable name. It imports the module, finds the application object, starts a local Ray instance if needed, deploys the application, and stays in the foreground. It serves on port 8000 by default, and Ctrl+C stops it.

A good answer adds why the constructor matters. Whatever you pass to bind goes to __init__, which runs once per replica. That is where you load a model, so the cost is paid at start-up and not on every request.

The commands to have ready

You are rarely asked to type these from memory, but you should recognise each and say what it does.

pip install -U "ray[serve]"
python -c "import ray; from ray import serve; print(ray.__version__)"
serve run module:app
serve build module:app -o serve_config.yaml
ray start --head
serve deploy serve_config.yaml
serve status
serve config
serve shutdown

In words: install with the serve extra, quoted so that zsh does not complain about the brackets; check the version; run in the foreground for development; generate a config file; start a head node; send the config to the cluster; check state; show the running config; remove everything. Mention that serve deploy returns as soon as the cluster accepts the request, so you must poll serve status to learn when the application is actually running.

The Python equivalents are serve.run(app, name=..., route_prefix=...), serve.status(), serve.delete(name) and serve.shutdown().

The classic traps and why they happen

Interviewers like to ask "what went wrong when you first tried this?" because the answers reveal real experience. These are the honest answers for most beginners.

Loading the model in the request method. The service works but every call is slow. The cause is that the model loads on every request instead of once. The fix is to load it in __init__.

The import path not importable on a cluster. It works on your laptop because the current folder is on Python's path, then fails on a cluster, because replicas are separate processes that do not have your file. The documentation states that the import path must be importable by Serve at runtime. The fix is a working_dir in the runtime_env, a container image, or a remote location.

Blocking code inside async def. A heavy synchronous call in an async method freezes the replica's event loop, so other requests queue behind it. The cause is that async methods share one loop. Keep async methods for waiting and use plain def methods for heavy work, which Serve runs in a thread pool.

Treating serve deploy success as "it works". The message only means the request was received. Replicas still have to start and models load.

Autoscaling that never scales. The default max_replicas is 1. A config that sets only min_replicas or target_ongoing_requests cannot scale up.

Using both num_replicas and autoscaling_config. They conflict. Choose one.

Old option names. Tutorials from before Ray 2.32 use max_concurrent_queries and target_num_ongoing_requests_per_replica. Current names are max_ongoing_requests and target_ongoing_requests, and the defaults changed too, to 5 and 2.0. Copying old code leads to errors or surprising behaviour.

Exposing the dashboard. Port 8265 allows running code on the cluster. Binding it to a public address is a serious security mistake.

Common interview questions

What problem does Ray Serve solve, and why not just use FastAPI?

A web framework like FastAPI is a good starting point for one small model on one machine. The trouble comes with scale. To use more cores you start more processes that each load the model. To use more machines you need load balancing and deployment tooling. You cannot scale one slow step of a pipeline without scaling the whole app, and sharing GPUs between workers is manual.

Serve gives you replicas, load balancing, per-component resources, composition and autoscaling while keeping your code in Python. It can even wrap a FastAPI app with @serve.ingress, so it adds to FastAPI rather than replacing it. My answer would be that FastAPI is the right choice until I need several replicas, shared GPUs or multi-step composition.

What is the difference between a deployment and a replica?

A deployment is the description: a class decorated with @serve.deployment. A replica is a running copy of it, a Ray actor. One deployment can have many replicas. Scaling means changing the number of replicas, not changing the deployment.

Where should I load a model, and why?

In __init__. It runs once when a replica starts, so the loading cost is paid once. Loading inside the request method would repeat it on every request. The model then lives in the replica's memory for later calls. A related point is that every replica holds its own copy, so memory use is the model size times the number of replicas.

How do you scale a deployment?

Two ways. For a fixed size, set num_replicas. For load-driven scaling, set autoscaling_config with min_replicas, max_replicas and target_ongoing_requests. Serve watches the average number of ongoing requests per replica and adds or removes replicas to hold that target between the bounds. Scale-up has a short delay and scale-down has a long one, 30 and 600 seconds by default, so brief lulls do not remove capacity. I would also mention that min_replicas: 0 saves money but causes a cold start on the first request.

How do you give a deployment a GPU?

Through ray_actor_options, for example {"num_gpus": 1}. Fractions such as 0.5 are allowed, so two replicas can share one card. These are logical resources used for scheduling. Ray does not enforce memory limits inside the fraction, so the model must fit its share.

How does one deployment call another?

By passing a bound deployment into another's bind call. The receiving class's constructor gets a DeploymentHandle. It calls await handle.remote(x) for __call__ or await handle.method.remote(x) for a named method. The call returns a DeploymentResponse, which is awaitable; from synchronous code you use .result(). This is how you build pipelines whose parts scale separately. Older material talks about deployment graphs and DAGDriver; that has been replaced by this approach.

What is the ingress, and how do I add FastAPI?

The ingress is the top deployment of an application, the one that receives outside requests. To get FastAPI routing and validation, create an app object, decorate the class with @serve.ingress(app) below @serve.deployment, and declare routes with the app's decorators. Pydantic models validate request bodies for you.

What does @serve.batch do?

It collects individual requests into a list so that a model can process them together, which is much more efficient on GPUs and vectorised code. The decorated method is async, receives a list and returns a list of the same length. You tune max_batch_size and batch_wait_timeout_s; larger batches and longer waits improve throughput at the cost of latency.

Why use a config file instead of serve run?

serve run is for development: it runs in the foreground and stops with the terminal. A config file is declarative and lives in Git. You generate it with serve build, send it to a cluster with serve deploy, and change scaling by editing YAML rather than code. In production on Kubernetes, the same config is pasted into the serveConfigV2 field of a KubeRay RayService.

How do you debug a deployment that will not become healthy?

Start with serve status. If a deployment is stuck deploying or marked unhealthy, read the replica logs under /tmp/ray/session_latest/logs/serve/. The usual causes are an exception in the constructor, a resource request the cluster cannot meet, or an import that fails on the replica. Then reproduce outside Serve by importing the class in a Python shell and calling the method directly.

Is Ray Serve secure by default?

No. The HTTP endpoints have no built-in authentication, so I would put a gateway or ingress with authentication in front, or add a dependency in FastAPI. The Ray dashboard on 8265 can run code, so it must stay private. Secrets belong in the environment of the replica or in a secret manager, not in committed config.

Which Python versions and platforms are supported?

Current Ray supports Python 3.10 to 3.12, with 3.13 in beta, on Linux, on macOS with Apple Silicon, and on Windows in beta. Multi-node clusters on Windows are experimental, so for learning on Windows I would use WSL2. Serve is installed through pip install "ray[serve]" and shares its version with Ray; the current line at the time of writing is 2.58.

What changed recently that affects code?

The renames in 2.32 are the main ones: max_concurrent_queries became max_ongoing_requests, with the default changing to 5, and target_num_ongoing_requests_per_replica became target_ongoing_requests, with the default changing to 2. The smoothing factor options were replaced by upscaling_factor and downscaling_factor. Handles are DeploymentHandles now, and Pydantic version 2 is expected.

When would you not use Ray Serve?

For a single small model on one machine, or very low traffic, where a plain FastAPI service is simpler. Serve means running a Ray process and understanding its concepts, which is a real cost. It earns that cost with replicas, GPU sharing, composition or LLM serving. For comparison, I could name alternatives from the same catalogue such as BentoML, KServe or Triton, each with different strengths.

How would you serve a large language model?

Ray Serve has an LLM module, ray.serve.llm, that provides OpenAI-compatible endpoints on top of engines like vLLM, configured with LLMConfig. It needs a GPU in practice. For a beginner's answer, I would say I know the module exists and that the vLLM guide explains the engine, and I would not claim detailed experience I lack.

How to talk about your own project

Whatever question comes, ground your answer in something you built. A strong short story has four parts: what the service did, how many deployments it had and why you split them, one thing that broke, and how you found it. For example, you might say that your sentiment service had a cleaner, a classifier with two replicas and a FastAPI ingress, that it stayed in a deploying state because you asked for a GPU on a laptop, and that serve status plus the replica log showed the resource wait. Interviewers remember a specific failure and its diagnosis far longer than a list of features, and it shows that you read output instead of guessing.

Questions to ask the interviewer

Asking good questions shows you think about the system, not only the code. Ask whether they run Serve on Kubernetes with KubeRay or on virtual machines. Ask how they handle model loading time and cold starts. Ask how traffic is authenticated before it reaches Serve, and where the cluster runs, since many employers in the Gulf and Egypt have data-residency requirements for the region.

Sixty-second self-test

Cover the answers and say each one aloud.

1. What does @serve.deployment do, and does it start anything?
2. What are the four nouns, in one sentence each?
3. What does Model.bind(Preprocessor.bind()) give Model's constructor?
4. Why load the model in __init__?
5. What are the two ways to set the number of replicas, and why can you not use both?
6. What does serve deploy returning successfully actually tell you?
7. Name the two commands you run first when a deployment will not start.
8. What is the default max_replicas, and what does that mean for autoscaling?
9. What are the old and new names for the per-replica request limit?
10. Name two reasons you might choose plain FastAPI instead.

If you hesitated on more than two, reread the matching section of the detailed guide and run the example again. Having run it yourself is what interviewers can hear in your answers.

This file collects what beginners actually get wrong with Ray Serve, the habits that prevent most of those mistakes, and a checklist to run through before you call a service done. Each mistake follows the same shape: the symptom you see, the cause underneath, and the fix.

The mistakes beginners actually make

1. The service works but every request takes seconds

Symptom. The first request and every later one are slow, even for a tiny input. CPU is busy and the model seems to be doing far more work than the request needs.

Cause. The model is loaded inside the request method (__call__ or a route), so each request pays the loading cost. Beginners copy a notebook where loading and prediction sit in the same cell.

Fix. Move the load into __init__. It runs once per replica, so the cost is paid at start-up. Check by sending ten requests and confirming that only the first replica start-up is slow.

2. pip install ray[serve] fails with "no matches found"

Symptom. In zsh, the default shell on macOS, the command prints zsh: no matches found: ray[serve].

Cause. Square brackets are glob characters in zsh, so the shell tries to expand the word before pip sees it.

Fix. Quote it: pip install -U "ray[serve]". Quoting works in every shell, so make it a habit.

3. serve: command not found

Symptom. The install seemed to succeed, but the shell cannot find serve.

Cause. The serve program is installed into the virtual environment's bin folder, and the environment is not active in this terminal. Opening a new terminal tab is the usual way to lose it.

Fix. Activate the environment again with source .venv/bin/activate, then rerun. If you want to be sure which interpreter you are using, run python -c "import ray; print(ray.__file__)" and read the path.

4. It works with serve run but fails on a cluster

Symptom. A deployment that ran perfectly on your laptop fails on a cluster with an error that it cannot import your module.

Cause. The documentation says the import path, such as text_ml:app, must be importable by Serve at runtime. On a laptop the current folder is on Python's path. On a cluster, replicas are separate processes, perhaps on other machines, and your file is not there.

Fix. Ship the code with the application: set a working_dir in the runtime_env, bake the code into a container image, or use a remote location the cluster can read. For learning on one machine this does not arise, so do not be surprised when it appears later.

5. "Deployed successfully" but nothing answers

Symptom. serve deploy printed success, but curl gets a refusal or an error.

Cause. The success message means the cluster received the request. Replicas still have to start, models have to load, and resources have to be found.

Fix. Run serve status repeatedly until the application reports running. If it never does, open the replica logs in /tmp/ray/session_latest/logs/serve/. A deploying state for as long as the model takes to load is normal, not a fault.

6. The deployment waits forever for resources

Symptom. Status stays in a deploying state and the logs mention waiting for resources.

Cause. You asked for more than the machine has: four replicas with one CPU each on a two-CPU laptop, or num_gpus: 1 on a machine without a GPU. Ray treats these numbers as logical resources and will not start a replica it cannot place.

Fix. Lower num_replicas or the per-replica request, for example {"num_cpus": 0.5}, or add capacity. On a laptop without a GPU, delete the GPU request entirely.

7. A blocking call freezes a replica

Symptom. One replica answers one request at a time, other requests look like timeouts, and CPU use is not high.

Cause. A long blocking call (a heavy model pass, time.sleep, a synchronous network call) sits inside an async def method. Async methods share the replica's event loop, so a blocking call stops everything else on that replica.

Fix. Use async def only when you await something. Put heavy synchronous work in a plain def method, which Serve runs in a thread pool, or use async-friendly libraries. Remember request.json() is the usual reason a method must be async.

8. Autoscaling never adds replicas

Symptom. You configured autoscaling, hammered the service, and the replica count stays at one.

Cause. The default max_replicas is 1. If you set only min_replicas or target_ongoing_requests, the ceiling is still one. The other common cause is waiting too little: scale-up waits 30 seconds by default before acting.

Fix. Set max_replicas explicitly. Then keep the load running longer than the upscale delay and watch serve status.

9. An error about setting both num_replicas and autoscaling_config

Symptom. The deployment refuses to start and the message complains about the two options.

Cause. They are two ways of answering the same question: a fixed number or a number that follows load.

Fix. Pick one. If you want a fixed number, delete the autoscaling block. If you want it to follow load, delete num_replicas.

10. Option names from old tutorials do not work

Symptom. An unknown-option error or surprising behaviour from a copied snippet.

Cause. Names changed. max_concurrent_queries became max_ongoing_requests, and target_num_ongoing_requests_per_replica became target_ongoing_requests, around Ray 2.32. Defaults changed with them. The smoothing factors were replaced with upscaling_factor and downscaling_factor. Handles are DeploymentHandles now, not RayServeHandles, and the DAGDriver pattern has been replaced by .bind() composition.

Fix. Check the date of every tutorial. Prefer the official documentation for the version you installed, and run python -c "import ray; print(ray.__version__)" so you know which one that is.

11. Memory runs out with several replicas

Symptom. A replica dies unexpectedly, or the machine slows to a crawl, once you raise the replica count.

Cause. Each replica holds its own copy of the model. Four replicas of a 2 GB model need about 8 GB, plus overhead. The operating system may kill a process that uses too much memory.

Fix. Budget memory as model size times replicas. Reduce replicas, use a smaller model, or ask for a memory allowance in ray_actor_options so scheduling reflects reality.

12. A certificate error while installing or downloading models

Symptom. SSLCertVerificationError or a similar message from pip or a model download.

Cause. A network gateway intercepts HTTPS and Python does not trust its certificate. Inside Serve, the same problem hits runtime_env installs and downloads that happen on the replica.

Fix. Make the tool trust your organisation's root certificate. Do not disable verification. Ask your IT team for the certificate file and point the environment at it.

13. Exposing the dashboard to make deploying easier

Symptom. serve deploy -a http://<ip>:8265 from another machine is refused, so you open the port widely.

Cause. The dashboard listens on localhost by default for good reason: it can run arbitrary code on the cluster.

Fix. Keep it private. Use an SSH tunnel, a VPN, or a deployment pipeline that runs inside the same network. Never place 8265 on the public internet.

14. Secrets in the config file

Symptom. An API key appears in serve_config.yaml or in a runtime_env environment variable, committed to Git.

Cause. The config file is convenient, so it collects everything.

Fix. Read secrets from the replica's environment or a secret manager in __init__. Rotate any key that has been committed, even if you delete the line afterwards, because Git remembers it.

15. Confusing the service port with the dashboard port

Symptom. You open http://127.0.0.1:8265 expecting your API and see a dashboard, or you send curl to 8265 and get nothing useful.

Cause. Two different things listen on two ports. Serve answers your requests on port 8000 by default. The Ray dashboard, which also receives serve deploy requests, listens on 8265. Beginners mix them because both are on the same machine.

Fix. Send inference requests to 8000 and management traffic (deploys, the dashboard) to 8265. Write the two numbers on a sticky note until they are automatic.

16. Editing the code and expecting a running service to change

Symptom. You edit hello.py, call the service, and the old answer comes back.

Cause. The replicas imported your file when they started. Saving the file does not tell a running service to reload it.

Fix. Stop serve run with Ctrl+C and run it again, or redeploy through your config. Get into the rhythm of edit, stop, run, test. If you use a config file on a cluster, redeploy it with serve deploy and wait for status to settle.

17. A batched method returns the wrong number of results

Symptom. A method decorated with @serve.batch raises an error, or callers receive each other's answers.

Cause. The contract is strict: the method is async, receives a list, and must return a list of the same length in the same order. Returning a single value, a shorter list, or a reordered list breaks the pairing between requests and results.

Fix. Build the output with a list comprehension over the input list so that position i of the output answers position i of the input. Test the method with a short list by hand before relying on the grouping.

18. Forgetting to stop the cluster afterwards

Symptom. Your laptop fan spins and memory is low hours after you finished, or a new serve run behaves oddly because an old application is still deployed.

Cause. ray start --head launches background processes that keep running after you close the terminal. Applications deployed with serve deploy stay on that cluster until you remove them.

Fix. When you finish an experiment, run serve shutdown to remove the applications and ray stop to stop the Ray processes. Make the pair part of your clean-up, in that order, and run serve status first thing next time to see what, if anything, is still deployed.

Habits worth adopting on day one

Run the plain Python first. Before wrapping a model in Serve, import the class in a shell and call the method directly. If it fails there, Serve is not the problem. This habit saves more time than any other.

Keep __init__ heavy and the request method light. Load models, open connections and read files in the constructor. In the request path do only the work that depends on the request.

Use a virtual environment for every project. Ray brings many dependencies. A fresh environment per project keeps versions from fighting and makes a requirements.txt honest.

Pin the Ray version in your requirements file. Because Serve changes with Ray, ray[serve]==<version you tested> prevents the service from changing underneath you. Run pip index versions ray to see what exists, and record the version that worked.

Name your deployments and applications. @serve.deployment(name="classifier") and an explicit application name make serve status readable when you have three deployments instead of one.

Prefer FastAPI for the ingress. Validation, documentation and clear error messages come with it. Use Pydantic models for request bodies so malformed input is rejected before it reaches your model. The FastAPI guide is a useful companion.

Move settings into the config file early. Replica counts, resources and autoscaling bounds belong in YAML so that an operator can change them without editing code. Generate the file with serve build and commit it beside the code.

Check status before you test. After every deploy, run serve status first. Testing a service that is still starting wastes time and produces misleading errors.

Read the logs where they live. /tmp/ray/session_latest/logs/serve/ holds controller and replica logs. Opening them should be the second thing you do after serve status, not the last.

Measure before you tune. Send realistic traffic and watch status and latency before touching autoscaling or batching. Numbers chosen from measurement beat numbers chosen from instinct, and batching in particular has a latency price.

Start with a minimum of one replica. Scale-to-zero looks attractive and saves money, but the first request after a lull pays for a full model load. Use it only after you have measured the cold start and decided it is acceptable.

Containerise before you share. When a colleague must run your service, a container image with the dependencies baked in avoids a whole class of "works on my machine" failures. The Docker guide shows how, and the Kubernetes guide shows where it goes next.

Think about where it runs. If your service touches personal data for an employer in the Gulf or Egypt, data-residency rules may restrict which cloud region hosts the cluster, the logs and the model downloads. Settle that before launch, not after.

Three quick practice recipes

Recipe one: prove the replicas are real. Give a deployment num_replicas=3, return os.getpid() from __call__, and send twelve requests with a shell loop. You should see three different numbers. If you see one, check the status output; your replicas may still be starting.

Recipe two: feel the cost of loading. Write a service that sleeps three seconds in __init__ and returns instantly in __call__. Time the start-up and ten requests. Then move the sleep into __call__ and time ten requests again. The difference is mistake number one, made visible.

Recipe three: break it on purpose. Raise an exception in a constructor, ask for a GPU you do not have, and use a wrong import_path. For each, write down what serve status printed and which log line told you the cause. The next time it happens by accident, you will recognise it.

A short checklist

Before you call a Serve project finished, check each of these.

• The model loads in __init__, not in the request path.
• No long blocking call sits inside an async def method.
• Ray is installed in a virtual environment, and the version is pinned.
• Resources in ray_actor_options fit the machine, and memory is budgeted as model size times replicas.
• The ingress uses FastAPI with Pydantic validation, or the raw request is validated by hand.
• Replica counts, resources and autoscaling live in a YAML config produced by serve build.
• Autoscaling sets max_replicas explicitly and does not also set num_replicas.
• After every deploy you ran serve status and waited for the application to be running.
• No secret is written in the config file or committed to Git.
• The Ray dashboard on port 8265 is not reachable from the public internet.
• You tested the plain Python class outside Serve before blaming Serve.
• You know which Ray version you used, and you checked tutorials against it.
• You can explain deployment, replica, application and handle in one sentence each.

If every box is ticked, you have a service that other people can run, read and change without you in the room, which is the real goal of a first project.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. The guide is written against Ray 2.58.x. Serve ships inside the ray package and shares its version number, so check what you actually have with python -c "import ray; print(ray.__version__)" before trusting any default quoted here.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run on a real project in a few minutes.

How Serve works under the hood

Beginner gave you three nouns: deployment, replica, application. To predict how the system behaves under load you need the processes behind them, because every surprising behaviour at this level, such as slow scale-down, dropped requests or a stalled replica, traces back to one of them.

A Serve cluster runs three kinds of actor. A controller is a single, global, detached actor. It holds the desired state (which applications exist, how many replicas each deployment wants), reconciles reality towards it, makes autoscaling decisions and restarts replicas that die. It never sees your request payloads. Proxies are per-node actors that listen for HTTP (and gRPC if enabled), match the URL against each application's route_prefix, and forward the request to a replica. Replicas are the actors that run your class. Each has the resources you declared in ray_actor_options, and each is a separate process with its own copy of whatever your __init__ loaded.

The controller is off the request path. That matters for failure analysis: if the controller is briefly unavailable, traffic already flowing to existing replicas keeps flowing, while changes (scale-out, new versions) wait. The reverse is not true of the proxy and replicas, which are the data path.

Two limits decide whether a request is accepted or waits. The first is max_ongoing_requests, the number of requests a single replica will process at once. Since Ray 2.32 its default is 5; it used to be 100, and the old name was max_concurrent_queries. The second is the queue in front of replicas. A caller that finds every replica at its limit waits, and the optional max_queued_requests bounds how many may wait before new ones are rejected rather than left to pile up. Rejection is a feature: a fast failure is something a client can retry elsewhere, while a ten-minute queue is an outage that looks like a slow service.

This also tells you what "scaling" means. Serve does not count requests per second. It looks at how many requests are ongoing (being processed or queued) per replica, averaged, and compares that to a target. A model that takes 50 ms and a model that takes 5 seconds produce very different ongoing counts at the same request rate, and the autoscaler responds to that count, not to your traffic graph.

Replicas, concurrency and the event loop

Beginner showed a synchronous __call__. At work you will meet the question that quietly decides your throughput: is your handler def or async def?

Each replica runs an asyncio event loop. An async def handler runs on that loop. A plain def handler is dispatched to a thread pool so it does not block the loop. This gives you two rules that are easy to state and easy to violate.

Do not block inside async def. A time.sleep(2), a synchronous requests.get, or a heavy NumPy call inside an async handler freezes the loop for its duration. Every other request on that replica, including health checks, waits behind it. The symptom is a replica whose latency graph shows pauses that match no individual request, and a max_ongoing_requests limit that never seems to fill because nothing is being admitted.

Pick the handler type to match the work. If the handler awaits I/O, such as a database call, an HTTP call or another handle, use async def; one replica then juggles many requests while they wait. If the handler is CPU- or GPU-bound code from a library with no async API, which is most model inference, a sync def is fine, and concurrency is limited by threads and by the library's own locking. For PyTorch inference the practical advice is to keep max_ongoing_requests modest and scale replicas instead, because a GPU does not get faster when ten threads submit to it.

import asyncio
from ray import serve

@serve.deployment(max_ongoing_requests=10)
class Embedder:
    def __init__(self):
        import torch
        from transformers import AutoModel, AutoTokenizer
        self.tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
        self.model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2").eval()
        self.torch = torch

    async def __call__(self, text: str) -> list[float]:
        # Offload the blocking forward pass so the event loop stays free.
        return await asyncio.to_thread(self._embed, text)

    def _embed(self, text: str) -> list[float]:
        with self.torch.no_grad():
            batch = self.tok([text], return_tensors="pt", truncation=True)
            out = self.model(**batch).last_hidden_state.mean(dim=1)
        return out[0].tolist()

The pattern above, an async entry point that hands blocking work to asyncio.to_thread, keeps health checks and queue handling responsive while still letting you await handles inside the same method. The imports sit inside __init__ on purpose: the deployment class is pickled and shipped to replicas, and importing heavy libraries lazily keeps the driver, and the config-building step, light.

Resources interact with this too. ray_actor_options={"num_cpus": 1} reserves a CPU slot in Ray's scheduler; it does not limit what the process actually uses. A replica reserving one CPU that spawns eight threads of BLAS work will oversubscribe the node, and neighbours slow down. Set the reservation to match what the code really consumes, and cap library threads (for example OMP_NUM_THREADS) through the runtime environment when you pack several replicas on a node. Fractional GPUs follow the same logic: num_gpus: 0.5 lets two replicas share a card in scheduling terms, but nothing stops either from allocating all the GPU memory, so you must size models to fit together.

Composition with DeploymentHandle

Real inference APIs are rarely one model. They are a tokenizer, a retriever, a ranker and a generator, glued by business rules. Serve's answer is model composition: each stage is its own deployment, scaled and resourced independently, and the ingress calls the others through handles.

The mechanism is .bind() with deployment arguments. When you write Model.bind(Preprocessor.bind()), Serve sees that an argument is itself a bound deployment, deploys both, and passes your Model.__init__ a DeploymentHandle where you wrote the inner bind. You never construct handles by hand in application code.

from ray import serve
from starlette.requests import Request

@serve.deployment(num_replicas=2)
class Preprocessor:
    def __call__(self, text: str) -> str:
        return text.strip().lower()

@serve.deployment(ray_actor_options={"num_gpus": 0.5})
class Classifier:
    def __init__(self):
        from transformers import pipeline
        self.pipe = pipeline("sentiment-analysis", device=0)

    def __call__(self, text: str) -> dict:
        return self.pipe(text)[0]

@serve.deployment
class Ingress:
    def __init__(self, pre, clf):
        self.pre = pre      # DeploymentHandle
        self.clf = clf      # DeploymentHandle

    async def __call__(self, request: Request) -> dict:
        text = (await request.json())["text"]
        clean = await self.pre.remote(text)
        return await self.clf.remote(clean)

app = Ingress.bind(Preprocessor.bind(), Classifier.bind())

Calling handle.remote(x) does not run anything inline. It returns a DeploymentResponse, a future-like object. Await it from async code, or call .result() from synchronous code. Two properties make handles powerful. First, a response can be passed straight into another handle call without awaiting it, so Serve chains the stages and the intermediate value never has to travel back through your ingress. Second, handles route using the same replica-selection logic as the proxy, so each stage load-balances across its own replicas and respects its own max_ongoing_requests.

Fan-out is natural: create several responses first, then await them together.

import asyncio

async def score_all(self, text: str) -> list[dict]:
    responses = [self.clf.remote(text) for _ in range(3)]   # started immediately
    return await asyncio.gather(*responses)

For streaming, a handle can be asked to stream: handle.options(stream=True).remote(x) returns a DeploymentResponseGenerator you iterate, which suits token-by-token output from a language-model stage.

The design question at this level is where to cut. Every boundary costs a serialisation and a hop. Split when two parts need different scaling (a cheap CPU preprocessor in front of a GPU model), different hardware, or different dependencies. Do not split a three-line normalisation function into its own deployment: you pay latency and complexity for nothing, and you add another autoscaler to reason about.

Note too that the old deployment-graph style with a driver class has been replaced by this plain .bind() composition. If you meet code that imports a DAGDriver, it predates the current API and should be rewritten rather than ported.

Dynamic batching with @serve.batch

A GPU running one input at a time is mostly idle. Batching groups concurrent requests into one forward pass, trading a small wait for much higher throughput. Doing that by hand in a web service means queues, timers and awkward result routing; @serve.batch does all three.

You decorate an async method that takes a list and returns a list of the same length. Callers still call it with a single item and receive a single result; Serve holds each call briefly, collects them, runs your method once, and splits the output back to the right callers.

from ray import serve
from starlette.requests import Request

@serve.deployment(max_ongoing_requests=64)
class BatchedEmbedder:
    def __init__(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer("all-MiniLM-L6-v2")

    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.05)
    async def embed(self, texts: list[str]) -> list[list[float]]:
        vectors = self.model.encode(texts)
        return [v.tolist() for v in vectors]

    async def __call__(self, request: Request) -> list[float]:
        text = (await request.json())["text"]
        return await self.embed(text)

The decorator takes a handful of parameters. max_batch_size (default 10) is the upper bound on batch size. batch_wait_timeout_s (default 0.01) is how long the first request in a batch waits for company before the batch runs anyway. max_concurrent_batches (default 1) allows several batches in flight on one replica, which helps when your method awaits I/O but rarely when it holds a GPU. An optional batch_size_fn lets you measure batch size in something other than item count, for example tokens, so a few long inputs do not form an oversized batch.

Tuning is a latency-budget exercise. The wait timeout is added latency for the first request in each batch under light load, so keep it well below your latency target minus the inference time. The documentation suggests powers of two for max_batch_size, which suit how accelerators tile work. Raise the batch size until throughput stops improving or memory runs out, then lower the timeout until median latency is acceptable. Under heavy load batches fill instantly and the timeout does not matter; under light load the timeout is the whole story.

Batching and max_ongoing_requests interact. A batch can only be as large as the number of requests the replica has admitted. If max_ongoing_requests is 5 and max_batch_size is 16, batches will never exceed 5. Set the replica limit comfortably above the batch size, and remember that autoscaling's ongoing-request target now describes batches' worth of waiting callers, so give it headroom too.

Batched async generators are also supported for streaming workloads, which is the route to batched token streaming without writing your own scheduler. For full language-model serving, though, prefer the purpose-built engines described later.

Autoscaling in depth

Fixed replica counts waste money at night and fall over at noon. Replace num_replicas with an autoscaling_config and the controller adjusts the count from the average number of ongoing requests per replica. The loop is simple to describe: every metrics_interval_s replicas report their ongoing requests; the controller averages them over look_back_period_s; it divides total ongoing requests by target_ongoing_requests to get a desired replica count; and it applies the delays and factors below to avoid flapping.

The names upscale_smoothing_factor and downscale_smoothing_factor are deprecated in favour of upscaling_factor and downscaling_factor; the old target_num_ongoing_requests_per_replica is likewise replaced by target_ongoing_requests. Do not set num_replicas alongside autoscaling_config; Serve treats the two as mutually exclusive.

applications:
  - name: embed_app
    route_prefix: /
    import_path: embedder:app
    deployments:
      - name: BatchedEmbedder
        max_ongoing_requests: 16
        ray_actor_options:
          num_cpus: 2
        autoscaling_config:
          min_replicas: 2
          max_replicas: 10
          target_ongoing_requests: 8
          upscale_delay_s: 15
          downscale_delay_s: 300

How to choose the target? Start from what one replica can do. If a replica serves a request in 200 ms and you want roughly 80 percent utilisation, it should hold about 4 to 5 requests in flight on average. Then set max_ongoing_requests somewhere 20 to 50 percent above the target, as the documentation advises, so a burst has room before queueing begins. A target that equals the maximum leaves no slack: the autoscaler only reacts once replicas are already saturated.

The delays are asymmetric by design. Scale-up is quick (30 seconds) because under-provisioning hurts users; scale-down is slow (10 minutes) because loading a model again is expensive and traffic is bursty. If your replicas take three minutes to load weights, a short downscale delay will thrash: you remove a replica, traffic returns, and you pay the load time again while requests queue. Lengthen downscale_delay_s rather than shortening upscale_delay_s to compensate.

Scale-to-zero (min_replicas: 0) suits spiky, tolerant workloads such as internal tools used during office hours. The price is that the first request after idle waits for a replica to be scheduled, a container to be ready if nodes also scaled down, and a model to load. For user-facing endpoints, keep at least one replica warm.

Replica autoscaling only asks for resources; it does not create machines. If a new replica needs a GPU and no node has one, it sits in a pending state until the node autoscaler (the Ray autoscaler on VMs, or the KubeRay autoscaler plus your cloud's node provisioner on Kubernetes) adds capacity. That is two loops in series, and the second is slower. Capacity planning has to include node start time, image pull time and model load time, not only the Serve settings.

Configuration as the source of truth

Decorator arguments are fine for a notebook. In production you want the shape of the deployment, replica counts, resources, autoscaling, to live in a reviewable file that can change without rebuilding code. That file is the Serve config, and the CLI turns your Python application into one.

serve build text_ml:app -o serve_config.yaml
ray start --head
serve deploy serve_config.yaml
serve status

serve build imports the application and writes a YAML skeleton with every deployment and its current options. You edit it, commit it, and treat it as the declarative description of what should run. serve deploy submits it to a running cluster's dashboard address (port 8265 by default); to reach a remote head node, pass -a http://<head-ip>:8265. Most Serve CLI commands read RAY_DASHBOARD_ADDRESS for that address, while serve run and serve start use RAY_ADDRESS.

Two behaviours catch people. First, serve deploy returns as soon as the request is accepted. A success message means the config was received, not that replicas are healthy. Always follow with serve status, and in CI poll it until the application reports running or a timeout passes. Second, Serve config is declarative and idempotent: re-deploying the same file changes nothing, and removing an application from the list removes it from the cluster. That is exactly what you want from a pipeline, and a reason to keep the file complete rather than patching it with ad hoc commands.

The config has a top-level shape worth knowing:

proxy_location: EveryNode
http_options:
  host: 0.0.0.0
  port: 8000
applications:
  - name: text_app
    route_prefix: /
    import_path: text_ml:app
    runtime_env:
      pip: ["transformers", "torch"]
    deployments:
      - name: Summarizer
        num_replicas: 2
        max_ongoing_requests: 10
        ray_actor_options:
          num_cpus: 1

Note that applications: is a list. The older single-application form with import_path at the top level is superseded by this multi-application layout, and each application carries its own name and route_prefix. Multiple applications can share one cluster, which is convenient and also a trap: they share the same object store, dashboard and failure domain. We return to that in the security section.

Anything set in the config file overrides what the decorator said. That is deliberate, and it is also why "I changed the decorator and nothing happened" is a classic complaint: the YAML still wins. Pick one place to own each setting. A good split is that code owns what the code needs to function (the class, imports) while the config owns what operations tune (replicas, resources, autoscaling, timeouts).

To change a running model's behaviour without restarting replicas, use user_config with a reconfigure method. A deployment that defines reconfigure(self, config) receives the new value in place whenever the config changes, so you can swap a threshold, a prompt template or a feature flag in seconds instead of reloading weights.

from ray import serve

@serve.deployment(user_config={"threshold": 0.5})
class Scorer:
    def __init__(self):
        self.threshold = 0.5

    def reconfigure(self, config: dict):
        self.threshold = config["threshold"]

    def __call__(self, score: float) -> bool:
        return score >= self.threshold

Dependencies and runtime environments

On a laptop every deployment shares your virtual environment. On a cluster, replicas run on other machines, and the question "how do they get their packages and your code" has to be answered on purpose. Serve offers a runtime environment per application or per deployment, and the docs are blunt about its limits.

A runtime_env can install pip packages, set environment variables, and ship a working_dir. The two failure modes are both about import paths. The import_path in your config, say text_ml:app, must be importable on every node that runs the application. If it is not, you see an import failure when the app starts. Fix it with a working_dir that points to your code (a local path for a quick test, a remote URI for something repeatable) or, better, by putting the code into the container image the cluster runs.

Per-deployment runtime environments, set under ray_actor_options, allow two deployments in one application to use conflicting library versions: one on one release of a package and one on another. Use this sparingly. The price is that each distinct environment is built separately, so start times grow, and you now have to reason about which replica has which version.

applications:
  - name: multi_env
    route_prefix: /
    import_path: pipeline:app
    deployments:
      - name: LegacyScorer
        ray_actor_options:
          runtime_env:
            pip: ["scikit-learn==1.3.2"]
      - name: NewScorer
        ray_actor_options:
          runtime_env:
            pip: ["scikit-learn==1.5.2"]

The documentation's strongest advice is about what not to put in pip. Packages that compile from source during installation burn CPU on production nodes and can destabilise a cluster mid-deploy. For anything heavy, such as PyTorch, CUDA libraries or compiled extensions, bake the dependencies into the image and keep the runtime environment for light, pure-Python additions. The practical rule is: the image holds what is slow and stable; the runtime environment holds what is small and changes often.

Since heavy imports are slow, put them inside __init__ (a delayed import) rather than at module top level. Otherwise the process that builds the application, such as serve build on your CI machine, has to import PyTorch and every model library just to read a decorator, and it may not have the GPU stack at all.

Model weights deserve the same thought. Downloading from a model hub in __init__ makes every replica start depend on an external service and on bandwidth. For production, either bake weights into the image, mount them from a volume, or pull from your own storage in your own region. For teams in the Gulf and Egypt this is also a data-residency choice: pulling a model from a distant public hub on every scale-up is slower, and fetching it from object storage in your own cloud region keeps the artefact path inside your boundary.

Running on Kubernetes with KubeRay

The recommended production path is Kubernetes with KubeRay, an operator that manages Ray clusters as Kubernetes resources. Serve's piece is the RayService custom resource, which the documentation says automatically handles health checking, status reporting, failure recovery and upgrades. You describe two things in it: serveConfigV2, which is the Serve config you built above pasted in as a string, and rayClusterConfig, which is an ordinary RayCluster spec with a head group and worker groups.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: text-service
spec:
  serveConfigV2: |
    applications:
      - name: text_app
        route_prefix: /
        import_path: text_ml:app
        deployments:
          - name: Summarizer
            autoscaling_config:
              min_replicas: 1
              max_replicas: 5
  rayClusterConfig:
    rayVersion: "2.58.0"
    headGroupSpec:
      rayStartParams: {}
      template:
        spec:
          containers:
            - name: ray-head
              image: registry.example.com/text-ml:1.4.2
    workerGroupSpecs:
      - groupName: workers
        replicas: 1
        minReplicas: 1
        maxReplicas: 4
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: registry.example.com/text-ml:1.4.2

Treat that file as an outline, not a copy-paste target: field names such as the autoscaling options on the cluster side move between KubeRay releases, so validate against the KubeRay documentation for the operator version you run. What matters conceptually is the two kinds of upgrade.

An application-level change, where only the Serve config differs (replica counts, autoscaling values, a user_config), is applied in place by the controller. It is quick, it keeps the cluster, and it is the right route for tuning. A cluster-level change, where the image or the RayCluster spec differs, makes KubeRay build a new cluster, wait until it is ready, switch the Kubernetes Service to it, and only then remove the old one. That yields zero-downtime upgrades at the cost of temporarily running two clusters, so your quota must have room for both. On virtual machines without Kubernetes the docs give the same advice for significant code changes: bring up a new cluster, deploy to it, and move traffic, rather than editing a live one.

Two sizing habits matter on Kubernetes. First, keep Serve's replica autoscaling and the cluster's node autoscaling consistent: a max_replicas that needs more GPUs than the worker group's maximum can ever provide leaves replicas pending forever. Second, give pods realistic memory and CPU requests and limits that match the ray_actor_options reservations; Ray places by its own numbers, Kubernetes enforces its own, and mismatches show up as out-of-memory kills of whole pods.

Surviving failure has layers. Replicas that die are restarted by the controller; proxies and the controller are restarted too. The head node is the weak point, because it hosts the Global Control Store; surviving its loss needs GCS fault tolerance with an external Redis, which is its own topic in the Ray documentation. If an outage of the head is unacceptable, read that page before you promise anything to your stakeholders.

For the Kubernetes mechanics underneath, see the Kubernetes guide, and for packaging the image the cluster runs, see Docker.

Health, timeouts and backpressure

Failure handling is where "it works" becomes "it keeps working". Serve gives you four levers, and each answers a different question.

Health checks. Serve probes each replica periodically. A deployment can define a check_health method; if it raises, the replica is marked unhealthy and replaced. Use it for conditions that make the replica useless while the process lives: a loaded model that went stale, a lost connection to a feature store. The check frequency and timeout are settable per deployment (health_check_period_s, health_check_timeout_s). Keep the check cheap and local; a health check that calls a slow downstream turns that downstream's blip into a restart storm of your replicas.

from ray import serve

@serve.deployment(health_check_period_s=10, health_check_timeout_s=30)
class Model:
    def __init__(self):
        self.ready = True

    def check_health(self):
        if not self.ready:
            raise RuntimeError("model not ready")

Timeouts. A request that runs past request_timeout_s is cancelled and the client gets an error. Without a bound, a stuck call holds one of a replica's few ongoing-request slots forever. Set a timeout a little above your worst legitimate latency, and make it shorter than the timeout of whatever sits in front of Serve, so the failure is yours and explainable rather than a mysterious gateway disconnect.

Backpressure. When replicas are full, max_queued_requests decides whether new requests wait or get rejected. For latency-sensitive services, a small queue plus fast rejection lets clients retry another instance and keeps tail latency bounded. For batch-like workloads, a longer queue is acceptable. Pair this with autoscaling and you get a clean story: load rises, ongoing requests per replica climb, replicas are added after the upscale delay, and during that window excess requests are queued briefly or shed.

Graceful shutdown. When a replica is being removed, such as during a scale-down or a rolling update, Serve stops sending it new requests and waits for ongoing ones to finish, up to graceful_shutdown_timeout_s. If your requests can legitimately last two minutes (a long generation) and the timeout is the default, they are cut off. Align this setting with your longest request, and make sure your own handlers do not swallow cancellation.

If you meet an error you cannot place, remember that these four mechanisms are the usual suspects. A "request dropped" or 503 under load points at queue limits. A 408 or 504 on slow calls points at the request timeout. A replica restart loop points at the health check. An abrupt cut at deploy time points at graceful shutdown. Exact message wording varies by version, so read the one your cluster prints rather than matching on text from a blog.

Observability

You cannot tune what you cannot see. Serve gives you four windows: the dashboard, logs, request IDs and metrics.

The Ray Dashboard (port 8265) shows applications, deployments, replicas, nodes and logs, and is the fastest way to answer "which replica is stuck". The CLI mirrors it: serve status prints application and deployment states, and serve config prints the configuration the cluster is actually running, which is how you find out that the file in git and the file on the cluster have drifted.

Logs are written on each node under /tmp/ray/session_latest/logs/serve/, with separate files for the controller, proxies and each replica; the Python logger is named ray.serve. Since Ray 2.9 a LoggingConfig, settable per deployment or application, controls the encoding (plain text or JSON), the log level and whether access logs are written. JSON logs are worth turning on as soon as logs leave the machine, because a log pipeline can index fields instead of parsing text.

import logging
from ray import serve

logger = logging.getLogger("ray.serve")

@serve.deployment(logging_config={"encoding": "JSON", "log_level": "INFO"})
class Model:
    def __call__(self, text: str) -> str:
        logger.info("scoring request", extra={"chars": len(text)})
        return text

Every request carries an X-Request-ID. Accept the one your gateway sets, or let Serve generate it, and include it in the logs you write. When a user reports one bad answer, that identifier lets you follow the call from the proxy through each replica it touched. Without it you are matching timestamps.

Metrics are exposed in Prometheus format when you start Ray with --metrics-export-port, for example ray start --head --metrics-export-port=8080. Serve metrics carry a ray_serve_ prefix and cover request counts, errors, processing latency and replica counts per deployment; list the exact names from your own /metrics output instead of copying them from elsewhere, since they have been refined across releases. The signals worth a dashboard are request rate, error rate, latency percentiles per deployment (not only at the ingress), ongoing requests per replica against the autoscaling target, replica count against max_replicas, and queue depth. For general Prometheus and Grafana practice, see the Prometheus guide and Grafana guide; for traces across services, OpenTelemetry.

Two diagnostics deserve names. A deployment whose ongoing requests per replica sit at the target and whose replica count sits at the ceiling is saturated: raise the ceiling or make the model faster. A deployment whose latency is high while ongoing requests are low is slow per request, which no amount of scaling will fix; profile the handler, and for memory questions you can set RAY_SERVE_ENABLE_MEMORY_PROFILING=1.

Security boundaries

Serve's HTTP endpoints have no authentication of their own. Whatever reaches port 8000 reaches your model. That is not an oversight to patch in a hurry; it is a statement about where the boundary belongs. Authenticate in front of Serve with an ingress controller, an API gateway or a service mesh, or inside your FastAPI app with a dependency that checks a token. Rate limiting and quotas belong there too.

The more dangerous surface is the Ray Dashboard and Jobs API on port 8265. They can run arbitrary code on the cluster. Exposing them to an untrusted network is equivalent to handing out shell access. Keep them on a private network, never bind --dashboard-host=0.0.0.0 on anything reachable from the internet, and use Kubernetes network policies to limit who can talk to the head pod. Reaching serve deploy -a http://<head>:8265 from CI should happen over a private path.

Multiple applications on one cluster share an object store, a dashboard and a failure domain. Per-deployment runtime environments separate Python dependencies, not security. If two tenants must not see each other's data or affect each other's availability, give them separate clusters or namespaces rather than separate route_prefix values.

Secrets deserve a firm rule: do not write API keys or tokens into the Serve config or into runtime_env.env_vars in a repository. Mount them as Kubernetes Secrets, as environment variables or files, or fetch them from a secret manager, and read them in __init__. See Vault for a manager-based approach.

Testing Serve applications

Deployments are Python classes, which makes them easy to test if you design for it. There are three layers.

Unit tests on logic. Keep the model call and the business rules in plain methods or helper functions, and test them without Ray at all. The decorator does not prevent this: access the underlying class and instantiate it directly, or better, put the logic in a separate module the deployment calls. A test that needs a cluster to check a threshold comparison is a test that will not get run.

Application tests in a local instance. Start the app in-process with serve.run, send real HTTP requests or use the returned handle, then clean up with serve.shutdown(). These tests catch wiring errors: wrong bind arguments, missing routes, serialisation problems.

import pytest
import requests
from ray import serve
from pipeline import app

@pytest.fixture(scope="module")
def running():
    serve.run(app, name="test", route_prefix="/")
    yield
    serve.shutdown()

def test_sentiment(running):
    r = requests.post("http://127.0.0.1:8000/", json={"text": "Great product"})
    assert r.status_code == 200
    assert "label" in r.json()

Configuration tests. Parse your Serve config in CI and assert things that have bitten teams: max_replicas is set, autoscaling and num_replicas are not both present, max_ongoing_requests exceeds the autoscaling target, no secrets appear in env_vars. These are cheap and they catch the mistakes that cost money.

For model-quality checks, run a small fixed set of known inputs through the deployed endpoint in CI and compare against expected outputs or a metric threshold. For load behaviour, a short script that drives concurrency and records latency percentiles gives you a before-and-after for every tuning change; without it you are guessing. Heavy models make unit tests slow, so substitute a tiny stand-in model in the test environment while keeping the interface identical.

Performance tuning, in order of leverage

Tuning without a method is folklore. Work from the biggest effect to the smallest, and measure after each change.

1. Find where the time goes. Compare end-to-end latency at the client with processing latency inside the deployment. The gap is queueing and network. If it is large, the problem is capacity or configuration, not your model code.
2. Fix blocking calls in async handlers. One such call can cap a whole replica's throughput. It is the cheapest large win.
3. Batch. For GPU models, @serve.batch is usually the biggest throughput multiplier. Tune the size and the timeout against the latency target.
4. Right-size replicas. Match num_cpus and num_gpus to real use, and use fractional GPUs where models are small. Packing more useful work per node cuts cost directly.
5. Set the ongoing-request limits deliberately. max_ongoing_requests slightly above target_ongoing_requests; too low wastes capacity, too high hides saturation until latency explodes.
6. Cut start-up cost. Bake weights and dependencies into the image, keep a warm minimum, lengthen the downscale delay. Cold starts are the main cost of aggressive scaling.
7. Reduce serialisation. Large payloads moving between deployments are copied. Pass identifiers or small features between stages and keep bulk data where it is, or collapse two stages into one when they always run together.
8. Move to a purpose-built engine when the workload is large-language-model generation, where continuous batching and cache management matter more than anything generic.

Serve's LLM module, ray.serve.llm, packages that last option: an OpenAI-compatible application built from an LLMConfig, running on engines such as vLLM or SGLang, with autoscaling and routing handled by Serve. In recent releases it adds request routing that is aware of the engine's cache and token counts. Install it with pip install "ray[llm]" and expect a GPU. If you only need to run one language model with an OpenAI-style API and no custom composition, compare it with running vLLM directly; Serve LLM earns its place when you also want Ray's scheduling, multi-model routing, or composition with other deployments.

Debugging beyond the basics

When something is wrong, start with state, then logs, then isolation.

State first. serve status tells you whether an application is DEPLOYING, RUNNING, or has failed, and per deployment which replicas are healthy. A deployment that stays in deploying for minutes is waiting for something: resources, an image, a model download, or a constructor that is hanging. The dashboard's replica view usually names which.

Logs next. Replica logs under /tmp/ray/session_latest/logs/serve/ show constructor exceptions, which are the most common cause of a deployment that never becomes healthy. If the cluster says a deployment is waiting for resources, the message names what it needs, typically a GPU that no node offers. Read the resource numbers literally: a request for {'GPU': 1} on a CPU-only cluster will wait forever by design.

Isolation last. Reproduce the failure outside the cluster. Instantiate the deployment's class directly in a Python shell, call its methods with a failing input, and see whether the model code or the Serve wiring is at fault. If the direct call works but the endpoint fails, suspect serialisation, runtime environments or resource constraints.

A short symptom table covers most cases:

Version drift between the driver and the cluster is a quieter source of confusion. Serialisation and the Serve schema change across Ray releases, and Pydantic version mismatches (Serve uses Pydantic v2; v1 support has been dropped) show up as validation or import errors after an upgrade. Pin the Ray version in your image and in your CI, upgrade deliberately, and run your application tests on the new version before the cluster.

Neighbouring tools in the catalogue

Serve rarely stands alone. A few pairings come up repeatedly.

It sits on Ray, so cluster questions, such as object store sizing and placement, belong to that guide. If you serve an existing FastAPI service you can wrap it with @serve.ingress, and the FastAPI for ML guide covers the application layer. Experiment tracking and model registries live in MLflow; a common pattern is to load the registered model version in __init__, with the version supplied through user_config or an environment variable so that promotion is a config change. When Serve is not the right shape, compare BentoML, KServe, Triton and TorchServe. Serve favours Python-native composition and Ray's scheduler; KServe favours Kubernetes-native model serving; Triton favours GPU-optimised, framework-level serving. Drift and monitoring after deployment belong to Evidently.

For delivery, the config file and the image tag are your release artefacts. A sensible pipeline builds the image from a pinned base, runs the tests above, publishes an immutable tag, updates the RayService manifest in git, and lets Argo CD apply it. The rollback is a revert of one commit.

A final honest note: if one small model on one machine meets your load, a plain FastAPI service is simpler and you should use it. Serve pays for itself when you need several replicas, shared GPUs, multi-stage composition or LLM serving.

Putting it all together

Here is a compact, production-shaped service that uses most of this guide: a CPU preprocessing deployment, a batched GPU model, a thin FastAPI ingress with token authentication, and a config file that owns the operational settings.

import os
from fastapi import Depends, FastAPI, Header, HTTPException
from ray import serve

api = FastAPI()

def check_token(authorization: str = Header(default="")):
    expected = os.environ.get("API_TOKEN", "")
    if not expected or authorization != f"Bearer {expected}":
        raise HTTPException(status_code=401)

@serve.deployment
class Preprocess:
    def __call__(self, text: str) -> str:
        return " ".join(text.split())[:2000]

@serve.deployment
class Classifier:
    def __init__(self):
        from transformers import pipeline
        self.pipe = pipeline("sentiment-analysis", device=0)

    @serve.batch(max_batch_size=16, batch_wait_timeout_s=0.05)
    async def predict(self, texts: list[str]) -> list[dict]:
        return self.pipe(texts)

    async def __call__(self, text: str) -> dict:
        return await self.predict(text)

@serve.deployment
@serve.ingress(api)
class Gateway:
    def __init__(self, pre, clf):
        self.pre, self.clf = pre, clf

    @api.post("/classify", dependencies=[Depends(check_token)])
    async def classify(self, payload: dict) -> dict:
        clean = await self.pre.remote(payload["text"])
        return await self.clf.remote(clean)

app = Gateway.bind(Preprocess.bind(), Classifier.bind())

applications:
  - name: sentiment
    route_prefix: /
    import_path: service:app
    deployments:
      - name: Preprocess
        num_replicas: 2
      - name: Classifier
        max_ongoing_requests: 24
        request_timeout_s: 30
        ray_actor_options:
          num_gpus: 0.5
        autoscaling_config:
          min_replicas: 1
          max_replicas: 6
          target_ongoing_requests: 16
          downscale_delay_s: 600
      - name: Gateway
        num_replicas: 2

The release flow is: build the image with weights and dependencies baked in; run unit, application and config tests; push an immutable tag; run serve build or edit the committed YAML; deploy through the RayService manifest; poll serve status until running; and watch latency, error rate, queue depth and replica count against the ceiling for the first hour. Secrets arrive as environment variables from a Kubernetes Secret. The dashboard port is reachable only from inside the cluster network.

Where you are now

You can now read a Serve cluster the way an operator does: a controller keeping state, proxies accepting traffic, replicas doing the work, each limited by ongoing-request budgets that explain queueing, backpressure and scaling. You can compose pipelines with handles, batch for throughput, size autoscaling from a latency and capacity argument, own your configuration in a reviewed file, manage dependencies without destabilising nodes, run on Kubernetes with the right upgrade path, observe the system, test it, and keep its security boundary where it belongs.

What the Senior part adds is the platform view: failure modes at scale, which component breaks first, multi-tenancy and cost control across many teams, upgrade and migration strategy, incident playbooks, custom routing and autoscaling policies, and an honest account of where Serve stops and another tool should take over.

Sources

• Ray Serve overview
• Serve API reference
• Production guide
• Deploy on Kubernetes
• Deploy on VMs
• Handling dependencies
• Dynamic request batching
• Advanced autoscaling
• Monitoring
• Ray Serve LLM
• Installation
• Ray releases

Part two of three. At this level interviewers stop asking what a deployment is and start asking what you would do when a service misbehaves, and why you chose one setting over another. Answers are judged on reasoning: what you check first, what you rule out, and what you give up with each choice. Read the scenario section aloud at least once; the habit of narrating a diagnosis is the skill being tested.

Where this picks up

The model in one screen

A Serve cluster has a controller (one detached actor that holds desired state, autoscales and replaces dead replicas), proxies (per-node actors that accept HTTP or gRPC and route by route_prefix), and replicas (actors running your class). The controller is off the request path. Load is measured as ongoing requests per replica, not requests per second. Each replica admits at most max_ongoing_requests (default 5 since Ray 2.32; it was 100 as max_concurrent_queries). Autoscaling compares average ongoing requests to target_ongoing_requests (default 2.0) and moves the replica count with delays: 30 seconds to scale up, 600 to scale down.

Hold three sentences ready. "Serve scales on in-flight work, so a slow model and a fast model at the same request rate need different replica counts." "The config file is the source of truth in production, and serve deploy only means the request was accepted." "Replica autoscaling asks for resources; the node autoscaler provides them, and that second loop is slower."

Commands and settings to have ready

serve build text_ml:app -o serve_config.yaml
ray start --head --metrics-export-port=8080
serve deploy serve_config.yaml
serve deploy serve_config.yaml -a http://<head-ip>:8265
serve status
serve config

Settings you should be able to explain without notes: max_ongoing_requests, target_ongoing_requests, min_replicas, max_replicas, upscale_delay_s, downscale_delay_s, max_queued_requests, request_timeout_s, graceful_shutdown_timeout_s, user_config with reconfigure, and the batch parameters max_batch_size and batch_wait_timeout_s. If you name a setting, state its purpose and the failure it prevents, not only its default.

Scenario questions

Latency spikes appear at low traffic. What do you check first?

A strong answer separates "slow per request" from "queued". Start by comparing client-side latency with the deployment's own processing latency. If they match, the handler is slow and scaling will not help. If the client sees much more, requests are waiting.

Low traffic plus spikes is the signature of a blocked event loop. A synchronous call such as a blocking HTTP request, a time.sleep, or heavy computation inside an async def handler freezes the replica's loop, so everything queued behind it, including health checks, stalls. The fix is to make the handler a plain def (Serve runs those in a thread pool), or to hand the blocking part to asyncio.to_thread, or to use an async client.

Other candidates to mention briefly: a cold replica after scale-down, garbage-collection pauses on large objects, and noisy neighbours on a shared node. A weak answer says "add replicas" before establishing which of these it is.

The pipeline is slow. Where do you look?

Look per stage, not only at the ingress. Serve exposes per-deployment latency and request metrics, so find which deployment's processing time dominates. Then ask whether the wait is inside the stage (model time) or before it (queueing because its replicas are at max_ongoing_requests). A stage that is saturated needs more replicas or batching. A stage that is slow with idle replicas needs a faster model or a profile of its code.

Also count hops. Each boundary between deployments costs serialisation and a network call, so a pipeline split into many tiny deployments pays for it. Large payloads copied between stages are another common cost; pass identifiers or small features instead.

Requests are being rejected during a traffic burst. Why, and what do you change?

Under load every replica reaches max_ongoing_requests. New requests wait in a queue, and if max_queued_requests is set, requests beyond it are rejected, which clients see as an error such as a 503. This may be correct behaviour. The question is whether capacity arrives in time.

Check the autoscaler's timing: the upscale delay (30 seconds by default) means the first half-minute of a burst is absorbed by existing replicas and the queue. Options: raise max_ongoing_requests if replicas can really handle more concurrency; lower target_ongoing_requests so scale-up begins earlier; shorten the upscale delay; keep a higher min_replicas ahead of known peaks. Then check the second loop: if new replicas need GPUs that must be provisioned, node start time and image pull time dominate, and no Serve setting fixes that.

A strong answer also says that rejecting fast is a deliberate choice for latency-sensitive services, because clients can retry or fall back, and that you tune the queue length to how long a request is worth waiting.

A new version deployed, serve deploy printed success, and nothing changed. What happened?

The success message only means the request was received; the rollout continues asynchronously. Poll serve status. If the application sits in a deploying state, read the replica logs under /tmp/ray/session_latest/logs/serve/ for a constructor exception, and check for pending resource requests such as a GPU that does not exist.

If the status is running but behaviour is old, suspect the configuration precedence: the YAML overrides decorator arguments, so a value changed in code may be superseded by the file. Use serve config to see what the cluster is really running.

Replicas keep restarting. How do you diagnose it?

Three usual causes. A failing health check (a check_health method that raises, or one that calls a slow dependency and times out) causes the controller to replace healthy processes. An out-of-memory kill shows as an actor that died unexpectedly; check memory reservations, batch sizes and model size. A crash in the constructor loops forever, with the exception in the replica log.

Describe the order: status, then replica logs, then the dashboard's actor view for the death reason, then reproduce by instantiating the class directly in a shell. The point is to isolate model code from Serve wiring.

Cold starts hurt after quiet periods. What are your options?

Quantify first: cold start is node provisioning (if nodes scaled down), image pull, model download or load, and the first request's warm-up. Then choose. Keep min_replicas at one or more for user-facing endpoints. Lengthen downscale_delay_s so replicas outlive short lulls. Bake weights into the image or mount them to avoid downloading at start. Preload in __init__ and run a warm-up inference there. Reserve scale-to-zero for tolerant, spiky, internal workloads where the first-request delay is acceptable.

Two teams want to share one cluster. What do you say?

Serve supports multiple applications with separate names and route prefixes, and per-deployment runtime environments can isolate Python dependencies. But they share the object store, the dashboard and the failure domain, and the dashboard and Jobs API can run arbitrary code. If either team handles data the other must not see, or cannot tolerate the other's noisy load, they need separate clusters or namespaces. Mention quotas and per-application max_replicas for the shared case, and an authentication layer in front because Serve has none of its own.

How do you test a Serve application?

Three layers. Unit-test model and business logic without Ray by keeping them in plain functions. Run the application in a local test with serve.run, send real requests, and call serve.shutdown() afterwards. Test the config in CI for the mistakes that cost money: no max_replicas, both num_replicas and an autoscaling block, secrets in env_vars. Add a fixed-input quality check and a short load script that records latency percentiles. Substitute a tiny stand-in model in tests.

Trade-off questions

Batching or more replicas?

Batching raises throughput per GPU at the cost of added wait for the first request in each batch. More replicas raise capacity at linear cost and do not change per-request latency. For GPU models, batch first because an idle accelerator is the most expensive waste; then add replicas for the remaining load. The parameters are the lever: max_batch_size bounds the batch and batch_wait_timeout_s bounds the added latency, which must sit comfortably below the latency target minus the inference time. Under heavy load batches fill instantly and the timeout becomes irrelevant. Mention that max_ongoing_requests caps batch size in practice.

Scale-to-zero or a warm minimum?

Scale-to-zero saves money for spiky or internal services and costs the first caller a cold start. A warm minimum costs a baseline and gives predictable latency. Decide by who calls the service and what a slow first call costs them. A middle path is a minimum of one replica with a long downscale delay.

Runtime environment or a custom image?

A runtime environment is quick to iterate and fine for small pure-Python packages. It is slower to start, builds per distinct environment, and the docs warn that packages compiling from source at runtime burn resources and can destabilise a cluster. A custom image is slower to build but reproducible, fast to start and cacheable. The rule: the image holds what is heavy and stable; the runtime environment holds what is small and changes often. Add that a pinned Ray version in the image avoids driver and cluster drift.

Serve, KServe, BentoML or Triton?

Serve is Python-native, strong on composition, resource sharing and LLM serving, and uses Ray's scheduler. KServe is Kubernetes-native with standard inference protocols and scale-to-zero through its own stack. BentoML emphasises packaging and a developer experience for shipping services. Triton emphasises GPU-optimised, multi-framework serving. A good answer picks on requirements: need for composition and custom Python logic, existing Ray usage, Kubernetes-only operations, and latency-critical GPU throughput. It also says that for one small model, plain FastAPI is simpler.

Serve LLM or a dedicated engine directly?

ray.serve.llm wraps engines such as vLLM or SGLang with an OpenAI-compatible interface, Serve autoscaling and multi-model routing. If you need only one model behind an OpenAI-style API, running vLLM directly has fewer moving parts. Serve LLM pays off when you want Ray's scheduling, several models behind one endpoint, or composition with other deployments.

Rolling application update or a new cluster?

An application-level change (replica counts, autoscaling, user_config) is applied in place and is cheap. A change to the image or cluster spec triggers KubeRay to build a new cluster, switch the Service and delete the old one: zero downtime, but temporarily double the resources. On VMs, the docs recommend a new cluster for significant code changes. Say what you would check first: whether quota allows two clusters at once.

What a strong answer covers that a weak one misses

Concrete differences, each of which interviewers listen for:

• It separates the two kinds of slow. Slow per request versus queued. Different diagnosis, different fix.
• It remembers the event loop. Blocking calls in async handlers are the most common avoidable Serve bug.
• It treats serve deploy as asynchronous. Polls status in CI rather than trusting the message.
• It knows what is a reservation and what is a limit. num_cpus and num_gpus place replicas; they do not cap usage.
• It bounds cost. Always a max_replicas, and an alert when the system sits at it.
• It knows the security surface. No authentication on Serve endpoints, and a dashboard port that must never be public.
• It mentions versions. Names defaults that changed in 2.32 and checks the installed version before quoting any number.

Follow-up probes to expect

An answer often draws one more question. Prepare short replies to these.

"Why is the controller not a bottleneck?" Because it is off the data path. Proxies and replicas carry requests; the controller only reconciles state and autoscaling decisions. If it is briefly down, existing traffic continues and changes wait.

"What happens to in-flight requests during a scale-down?" The replica stops receiving new requests and finishes the ongoing ones up to the graceful shutdown timeout. Requests longer than that are cut off, so set the timeout above your longest legitimate request.

"How do you change a threshold without reloading the model?" Define reconfigure on the deployment and pass the value through user_config in the config file; replicas receive it in place.

"How would you authenticate callers?" In front of Serve through a gateway or ingress, or inside a FastAPI dependency. Serve has no built-in authentication. The dashboard on port 8265 is a separate, more dangerous surface and stays on a private network.

"How do you keep secrets out of the config?" Mount them from Kubernetes Secrets or a manager such as Vault and read them in __init__. Never commit them to the YAML or to env_vars.

"What do you alert on?" Error rate and tail latency per deployment, replicas pinned at max_replicas, ongoing requests per replica far above target, queue rejections, replica restarts, and pending resource requests. Tie each alert to an action.

"A GPU replica will not start." Read the status message: it usually says it is waiting for resources. Compare the requested GPU count and type with what nodes offer, then check the node autoscaler or the KubeRay worker group maximum. Remember that fractional GPUs share scheduling but not memory.

Questions to ask the interviewer

Good candidates ask about the team's real set-up, and it shows judgement. Useful ones: "Do you run Serve on Kubernetes with KubeRay or on VMs?" "How do you handle model weights, in the image or pulled at start?" "What is your target latency, and what batching do you use?" "How are releases rolled back?" "Who owns the dashboard's network exposure?" Each signals that you know where the operational risk lives.

Quick reference

Senior takes every one of these further, with a platform dimension. Which component fails first at scale and how to prove it. Custom request routing and autoscaling policies. Head-node fault tolerance and what survives it. Tenant isolation, cost governance across teams, and upgrade and migration strategy across Ray versions. Incident playbooks for saturation, memory kills and exposed dashboards. And an honest account of where Serve stops and another serving tool should take over.

Part two of three. At this level your service runs and answers requests, so the problems change character. They are no longer "why will it not start" but "why does latency spike at low traffic", "why did scale-down take ten minutes", and "why does it work on my laptop and not on the cluster". Start with the error table, then the practices ranked by leverage, the practice cards, and the recipes underneath.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work. Exact message text varies between Ray versions, so the first column describes what you see rather than quoting strings; read your own log for the precise wording.

Two of these deserve a longer reading because they cost the most time.

The blocked event loop. An async def handler runs on the replica's asyncio loop. Anything synchronous and slow inside it, a requests.get, a time.sleep, a large NumPy operation, holds the loop, and nothing else on that replica progresses meanwhile, including health checks. The cluster looks idle because ongoing requests are low, yet latency is high. The fix is not scaling. Move the blocking work off the loop or make the handler synchronous so Serve runs it in a thread pool.

Two autoscalers. Serve's replica autoscaler asks for resources. A node autoscaler (the Ray autoscaler on VMs, or KubeRay plus your cloud's node provisioning) supplies them. A replica pending for a GPU is waiting on the second loop, which includes machine start time and image pull. People tune the first loop for hours when the delay is in the second.

The practices that pay off most, in order of leverage

Why that order. The first two change what each replica can do, so they multiply everything else. Configuration ownership and image discipline remove whole classes of "works here, not there" failures. The cap, the timeouts, and authentication limit the damage when something else goes wrong. Testing keeps all of it from regressing. Observability underlies every item: without per-deployment latency and ongoing-request metrics you cannot tell which practice you need.

Why batching beats scaling first

A GPU running one input at a time spends most of its time waiting for work. Doubling replicas doubles cost and leaves each one equally idle. Batching raises utilisation inside the replica you already pay for. Do it first, then scale. The one caution is latency: the wait timeout adds delay to the first request in each batch under light load, so keep it well below your target minus inference time.

Why the image beats the runtime environment for heavy packages

A runtime environment builds on the node when the application starts. For PyTorch or compiled extensions this is slow, repeated per distinct environment, and the documentation warns that packages that compile from source at runtime burn resources and destabilise clusters. An image builds once, caches, and starts fast. The side benefit is reproducibility: the same bytes run in staging and production.

Why you should set a ceiling

Autoscaling reacts to load, including load caused by your own mistakes: a retry loop, a downstream outage that makes requests slow and therefore numerous in flight, or a runaway client. Without max_replicas the cluster scales, and the bill follows. With it, the failure becomes visible as saturation instead of expense.

Habits worth adopting

• Name the Ray version in every ticket. Defaults and names changed in 2.32 (max_ongoing_requests from 100 to 5, target_ongoing_requests from 1.0 to 2.0). Advice from older sources can be wrong for your version.
• Poll status after deploy. In CI, loop on serve status until running or a timeout. Never trust the deploy message.
• Delay heavy imports. Import libraries inside __init__ so building the application does not require the GPU stack.
• Read the live config. serve config before debugging; the file in git and the cluster can differ.
• Log the request ID. Include X-Request-ID in your own log lines so one bad answer can be followed across replicas.
• Keep one load script. Fixed concurrency, fixed inputs, p50 and p95 recorded. Every tuning change gets a before and after.
• Change one setting at a time. Otherwise nobody learns which one helped.
• Pin Ray in the image and in CI. Upgrade on purpose, run application tests on the new version first.
• Source weights from your own storage. Public model hubs on every scale-up are slow and add an external dependency; storage in your own region is faster and keeps the artefact path inside your data boundary.

Practice cards

1. Provoke the blocked loopWrite an async def handler that calls time.sleep(1). Fire five parallel requests and time them. Swap in await asyncio.sleep(1) and repeat. Write one sentence on why the totals differ.
2. Find your batching sweet spotBatch an embedding or classification model. Drive 50 concurrent requests at max_batch_size 1, 8, 16 and 32. Record throughput and p95 latency. Find where throughput stops improving.
3. Watch autoscaling workSet min_replicas: 1, max_replicas: 4, a short upscale delay. Apply load, watch serve status and the dashboard, then stop the load and time scale-down.
4. Make the YAML winBuild a config with serve build. Change a replica count in the file and a different one in the decorator. Deploy and use serve config to prove which value applied.
5. Break and read a constructorAdd a typo to __init__. Deploy, then find the exception by following serve status to the replica log.
6. Trigger backpressureSet max_ongoing_requests and max_queued_requests to 1. Flood the endpoint and observe rejections. Then relax the limits and watch them stop.
7. Hot-reconfigure a thresholdAdd reconfigure and a user_config value. Change it in the config file and confirm the replica picks it up without reloading weights.
8. Lock the front doorAdd a FastAPI dependency that requires a bearer token. Confirm an unauthenticated request is rejected. Check from another machine that port 8265 is not reachable.
9. Write a config testParse your Serve YAML in a pytest and fail if any autoscaled deployment lacks max_replicas, or if both num_replicas and autoscaling appear.

Operational recipes

Deploy and verify in a pipeline

serve deploy serve_config.yaml -a http://ray-head.internal:8265
for i in $(seq 1 30); do
  serve status -a http://ray-head.internal:8265 | tee status.txt
  grep -q "RUNNING" status.txt && exit 0
  sleep 10
done
echo "deployment did not reach RUNNING" >&2
exit 1

Adapt the check to the output format of your version, and treat a failure as a reason to read the replica logs. The address must be reachable from the pipeline over a private path.

See what is really running

serve config
serve status

Compare serve config with the file in git. Drift here explains "I changed it and nothing happened" more often than any bug.

Find replica logs

On each node, look in /tmp/ray/session_latest/logs/serve/. There are files for the controller, proxies and each replica. In a container, mount or ship this directory to your log system and turn on JSON logging so the pipeline can index fields.

from ray import serve

@serve.deployment(logging_config={"encoding": "JSON", "log_level": "INFO"})
class Model:
    def __call__(self, text: str) -> str:
        return text

Expose metrics

ray start --head --metrics-export-port=8080
curl -s http://localhost:8080 | grep ray_serve_ | head

List the metric names from your own output before building dashboards. Chart per-deployment latency, error rate, ongoing requests per replica against the target, replica count against the ceiling, and queue depth.

Check a replica without the cluster

from pipeline import Classifier

# .func_or_class gives the undecorated class
model = Classifier.func_or_class()
print(model("Great product"))

If calling the class directly works and the endpoint fails, suspect serialisation, a runtime environment or resources, not your model code. If that attribute is not available in your version, move the logic into a plain module and import it in both places.

Add a warm-up

@serve.deployment
class Model:
    def __init__(self):
        self.model = load_model()
        self.model("warm-up input")   # first call pays lazy initialisation here

Warming inside __init__ moves the slow first inference out of a user's request and into replica start-up, where the health system expects it.

Align graceful shutdown with your longest request

Set graceful_shutdown_timeout_s above the longest legitimate request, so a scale-down or rolling update does not cut off long generations. Make sure your handlers do not swallow cancellation.

Pick an upgrade path on Kubernetes

For a change only in serveConfigV2 (replica counts, autoscaling, user_config), expect an in-place update. For a new image or cluster spec, expect KubeRay to create a second cluster, switch the Service and remove the old one, so check quota for two clusters first. See the Kubernetes guide and Docker guide for the mechanics around it, and Argo CD to apply manifests from git.

Triage in two minutes

1. serve status: is the application running, deploying or failed?
2. Dashboard: which replica or node is the odd one out?
3. Replica log: constructor exception, out-of-memory, or a resource wait message?
4. Compare client latency with deployment processing latency: queued or slow?
5. serve config: does the live config match the file?
6. Reproduce the class directly in a shell to separate model code from Serve wiring.

How to read a saturated service

Saturation has a recognisable shape, and learning it saves hours. Replica count sits at max_replicas. Ongoing requests per replica hover at or above the autoscaling target. Latency percentiles climb together while the error rate stays low, until queues fill and rejections begin. At that point there are only three honest moves: raise the ceiling if budget and downstream systems allow, make each request cheaper through batching or a lighter model, or shed load deliberately with a smaller queue and a clear client-side retry policy.

Contrast that with a service that is merely slow. Replica count is low, ongoing requests are below target, and latency is high anyway. Nothing is queued; each request simply takes long. Scaling cannot help, because the autoscaler sees no pressure and would be right to do nothing. Profile the handler, check whether a downstream call dominates, and look for a blocked event loop. Teams that cannot tell these two shapes apart tend to add replicas to a slow service, pay more, and see no change.

A third shape is the oscillating service. Replicas rise, fall, and rise again within the same hour. That points at a downscale delay shorter than a model's load time, or at a load pattern with short bursts and a metrics window that reacts to each one. Lengthen the downscale delay first, then consider the look-back window and the aggregation function. The goal is for capacity to follow the trend of the load rather than every ripple in it.

Reviewing a pull request that touches Serve

When a colleague changes a deployment, a short review list catches most expensive mistakes. Does any async def call something that blocks? Does the change set max_replicas, and is it justified against budget and downstream capacity? Is max_ongoing_requests above both the autoscaling target and any batch size? Are timeouts ordered from the outside in? Did a new heavy package land in a runtime_env instead of the image? Does the config avoid secrets, and does a test cover the new behaviour? Is the Ray version unchanged, or deliberately bumped with tests run on it? Asking these five or six questions every time is worth more than any single clever optimisation.

Checklist before you call it production-ready

• Ray version pinned in the image and CI, and the guide's defaults checked against it.
• Weights and heavy dependencies baked into the image, with weights from storage you control.
• Heavy imports delayed into __init__; a warm-up inference in the constructor.
• No blocking calls inside async def handlers.
• Batching on GPU models, with size and timeout tuned and recorded.
• max_ongoing_requests above target_ongoing_requests, and above max_batch_size.
• max_replicas set, with an alert at the ceiling; min_replicas chosen on purpose.
• request_timeout_s, max_queued_requests and graceful_shutdown_timeout_s set and ordered sensibly.
• Health check cheap, local and meaningful.
• Config file in git, deploy verified by polling status, live config compared to the file.
• Authentication in front of Serve; port 8265 private; secrets from a secret store, never in the YAML.
• Metrics and JSON logs flowing, with X-Request-ID in your own log lines.
• Tests at three layers, plus a load script with recorded numbers.

Senior takes every one of these further, with a platform dimension. Failure modes at scale and which component breaks first. Custom request routing and autoscaling policies. Head-node fault tolerance and what survives its loss. Tenant isolation, cost governance and quotas across teams. Upgrade and migration strategy across Ray versions, and incident playbooks for saturation, memory kills and exposed dashboards. Plus the honest question of when to move to another serving tool such as KServe or Triton.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when Ray Serve is a platform that other teams deploy onto, rather than a library you call from your own project.

Up to this point the questions were about making a deployment work and making it fast. From here they change: what happens to forty applications when one node dies, who is allowed to reach port 8265, how a team that has never heard of an actor gets a model into production without paging you, and how you upgrade a cluster that carries someone else's revenue.

A note on versions. Ray Serve ships inside the ray package and shares its version number; there is no separate Serve release. This guide was checked against the Ray 2.58 line. Run pip index versions ray before you pin anything, and read the release notes of the version you are moving to, because several Serve defaults changed between 2.x releases and the changes are quiet.

Where this picks up

I will begin with the architecture, because every failure in this track is a failure of one specific component, and you cannot triage what you cannot name.

The architecture, and what each part costs you when it dies

You know the nouns: deployments, replicas, applications. As an owner you also need the machinery behind them. A Serve instance sits on a Ray cluster and has three kinds of long-lived process.

The controller is a single global detached actor. It owns the desired state: which applications exist, how many replicas each deployment wants, what the autoscaler decided. It does not sit in the request path. This matters more than any other fact about it. If the controller is down, requests that already have a route keep flowing, but nothing can change: no deploys, no scale-up in response to load, no replacement of a replica that has just died. Teams describe this as "the cluster froze", and it is a good phrase, because the data plane keeps working while the control plane is gone.

The proxies are per-node actors that accept HTTP and gRPC traffic and forward it to replicas. They are in the request path, so a proxy failure is user visible, but there are several, and a node-level load balancer in front of them routes around a dead one. The proxy_location setting in the config file decides where proxies run, with options for every node, the head node only, or none. Running them on every node gives you the most capacity and the most places to secure; running them only on the head node concentrates all ingress on one machine. For a shared platform, the usual choice is every node behind a load balancer, because it removes a single ingress bottleneck.

Replicas are Ray actors, each holding one copy of a deployment's state, which for ML usually means a loaded model. A replica that dies is restarted by the controller. The restart is cheap for a small function and expensive for a model: the replacement has to schedule, import, load weights, and pass its health check before it takes traffic. You should know the load time of every tenant's model, because that number is your real recovery time, not the few milliseconds the diagram implies.

Underneath all of this is the Ray cluster itself, and its head node runs the global control store. The guide-level summary is that Serve's own components recover on their own, but a lost head node is a different class of event from a lost worker, and surviving it requires fault-tolerant configuration of the cluster's control store. The Kubernetes production page describes what the KubeRay operator handles for you; for the exact head-node recovery options of your Ray version, read that page rather than relying on this paragraph.

Here is the failure table I keep next to the runbook.

Notice the pattern in the middle column. The first three rows are capacity events and the last two are control events. Capacity events are noisy and self-healing; control events are quiet and dangerous, because the data plane hides them until a second failure arrives. A good alert design pages on control-plane health even when the dashboards of traffic are green.

The request path and the contract it implies

A request is admitted by a proxy, routed to a replica of the ingress deployment, and possibly forwarded through DeploymentHandle calls to other deployments. Every replica accepts at most max_ongoing_requests concurrent requests. Since Ray 2.32 the default for that setting is 5, down from 100, and the old name was max_concurrent_queries. Anyone carrying an old config forward should check which name it uses, because a deprecated name quietly means the default no longer matches the intent.

That one number is a platform contract in disguise. It says how much concurrency a replica is willing to hold, and by implication how much of a spike the replica absorbs versus how much waits upstream. The docs advise setting it 20 to 50 percent above the autoscaling target, so the autoscaler sees load building before replicas are saturated. If the two are equal, a replica fills exactly when the autoscaler is first asked to act, and latency has already degraded.

As an owner you decide, per tenant class, what the policy is. A latency-sensitive service wants a low ceiling and fast scale-out, so that excess requests wait where the autoscaler can see them. A throughput service that batches wants a higher ceiling so batches can fill. Put the decision in a template, not in each team's head.

The failure that bites most often at this level is not on the autoscaler at all. It is the blocked event loop. Async deployment methods run on the replica's event loop; a blocking call inside an async def stalls every request that replica is holding, including health checks. Sync methods, by contrast, run in a thread pool on the replica, which is safe for blocking code but limited by the pool. A tenant who writes async def and then calls a blocking client library has built a replica that looks alive and serves nothing. You will see it as replicas that pass until traffic arrives, then time out together.

import asyncio
from ray import serve

@serve.deployment(max_ongoing_requests=10)
class Tenant:
    async def bad(self, x):
        import time
        time.sleep(2)               # blocks the event loop: every in-flight request stalls
        return x

    async def good(self, x):
        await asyncio.sleep(2)      # yields; other requests proceed
        return x

    def also_fine(self, x):         # sync: runs in a thread, blocking is contained
        import time
        time.sleep(2)
        return x

The review rule that follows is simple enough to automate in a linter: no synchronous I/O or heavy compute inside an async def handler. If a model call is CPU or GPU bound and blocking, the handler should be sync or should hand the work to a thread explicitly.

Two further contract items belong here. First, @serve.batch requires an async method that takes a list and returns a list of the same length. A tenant who returns a different length breaks every request in the batch, so test it. Second, timeouts: request_timeout_s is a config-level setting, and a platform without a default will eventually have a request that waits forever. Set a default and make tenants raise it with a reason.

Scaling: which component breaks first

Serve scales at two layers, and the skill is knowing which one is the bottleneck for a given load shape.

The first layer is replica autoscaling. The controller watches the average number of ongoing requests per replica and moves the replica count toward target_ongoing_requests, which defaults to 2.0 (it was 1.0 before 2.32). It is bounded by min_replicas and max_replicas, both of which default to 1, so an autoscaling config that names no maximum cannot scale at all. Delays smooth the signal: upscale_delay_s defaults to 30 and downscale_delay_s to 600. The scale-down delay of ten minutes is deliberate, because model replicas are expensive to recreate, and it means a bursty workload will hold peak capacity for a while after the burst. That is a cost, not a bug.

The second layer is node autoscaling: the Ray autoscaler, or the KubeRay autoscaler, adds machines when replicas cannot be placed. Replica autoscaling asks for resources; node autoscaling provides them. Between the two sits the slowest step in the whole chain, which is getting a GPU node, pulling a large image onto it, and loading weights. For a GPU tenant, this is minutes, and replica autoscaling's thirty-second reaction time is irrelevant against it.

So what breaks first? In order of how often I see it:

1. Node capacity. Replicas want resources the cluster does not have. They sit pending, and the symptom is a deployment stuck deploying with a message about waiting for resources. The autoscaler may be unable to find a node type with the requested GPU, which is a catalogue problem and not a Serve problem.
2. Cold start. Capacity exists but model load dominates. Scale-to-zero makes this the first-request experience.
3. A single hot ingress deployment. All traffic passes through one ingress replica set; if the ingress does heavy work, it saturates before the models behind it. Keep the ingress thin.
4. The controller. In very large deployments, with many applications and many replicas, controller work grows. Treat this as a reason to split clusters well before it hurts, and measure it rather than assume it.
5. The object store. Large payloads passed between deployments consume shared memory; a tenant who pipes images through five stages has found the limit. Ray wants roughly 30 percent of RAM for the object store by default, and in containers the shared memory size must allow it.

Autoscaling defaults you set once for everyone

Autoscaling is where platform teams earn their keep, because a default applied well prevents a hundred separate mistakes. These are the fields, with defaults from the autoscaling documentation.

Two deprecations to scan for in inherited configs: upscale_smoothing_factor and downscale_smoothing_factor were replaced by upscaling_factor and downscaling_factor, and target_num_ongoing_requests_per_replica became target_ongoing_requests. Do not combine a fixed num_replicas with an autoscaling_config on the same deployment; the platform validator should reject it before the controller does.

The reasoning behind the numbers is queueing, not Ray. The target is the concurrency you want each replica to run at. If a replica handles a request in 200 ms and you want it about 80 percent busy at one request at a time, the target belongs near 1; if a model batches eight requests efficiently, the target belongs far higher. The right way to choose is a load test that records latency against concurrency for one replica, finds the knee, and sets the target below it. Teams that copy another team's numbers inherit another model's knee.

applications:
  - name: ranker
    route_prefix: /ranker
    import_path: ranker.app:app
    deployments:
      - name: Ranker
        max_ongoing_requests: 12
        autoscaling_config:
          min_replicas: 2
          max_replicas: 10
          target_ongoing_requests: 8
          upscale_delay_s: 15
          downscale_delay_s: 900
        ray_actor_options:
          num_cpus: 2
          memory: 4294967296

Notice max_ongoing_requests at 12 against a target of 8: fifty percent above, inside the documented 20 to 50 percent guidance. The downscale delay is longer than the default because this model takes a while to load, and an extra five minutes of idle replicas is cheaper than a reload during a traffic dip.

One more control belongs to the platform rather than the tenant: a ceiling. max_replicas is a promise the cluster must be able to keep, or a loud failure waiting to happen. Sum the declared maximums across tenants and compare to real capacity; if the sum is larger, you have overcommitted, and you should decide on purpose whether that is acceptable.

Configuration as the source of truth

At Beginner level you ran serve run. At Mid you learned serve build and serve deploy. At this level the question is who is allowed to change a running cluster, and the answer should be: a reviewed commit, rolled out by automation.

The Serve config file is the declarative description of the whole cluster: HTTP options, proxy placement, and a list of applications, each with a name, route prefix, import path, runtime environment, and per-deployment overrides. The production guide's workflow is serve build text_ml:app -o serve_config.yaml, then serve deploy serve_config.yaml, then polling serve status. Two details matter for automation. First, a successful serve deploy only means the request was received; the rollout proceeds asynchronously, so a pipeline that stops at the exit code will report success for a deployment that later fails. Poll status until the application is running or fails. Second, serve deploy against a remote cluster takes -a http://<head>:8265, which means your pipeline is talking to the dashboard port. Think about that address before you open a firewall rule for it; the security section returns to it.

On Kubernetes, the recommended production path is KubeRay and its RayService resource. A RayService has two key fields: serveConfigV2, into which you paste the Serve config, and rayClusterConfig, the cluster specification. The docs describe the operator as handling health checking, status reporting, failure recovery, and upgrades. That makes the Git repository the interface: a team changes a YAML file, a reviewer approves it, and a GitOps controller applies it. Pair it with Argo CD and Helm if you already run them, and the Serve config becomes a rendered value rather than a hand-edited blob.

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: serving-platform
spec:
  serveConfigV2: |
    proxy_location: EveryNode
    applications:
      - name: ranker
        route_prefix: /ranker
        import_path: ranker.app:app
  rayClusterConfig:
    # head group and worker groups: image, resources, replicas
    rayVersion: "2.58.0"

Treat that as a skeleton, not a complete manifest: the worker group specification is long and depends on your cluster. The point is the split between the two fields, because they map to the two upgrade paths in the next section.

A governance rule worth writing down: tenants own their application block, and the platform owns everything else. Put each tenant's application in its own file or its own pull request path, with code owners set so that a tenant approves changes to its block and the platform team approves changes to cluster-level settings. Then a change to proxy_location cannot slip in through a tenant's pull request.

Upgrades and migrations

There are two kinds of change, and they have different blast radii.

An application-level change alters only the Serve config: replica counts, autoscaling parameters, a new application, a new version of application code that is available to the cluster. It applies in place and is lightweight. A cluster-level change alters the Ray cluster: a new container image, new Ray version, different node resources. With KubeRay's RayService, the docs describe the operator starting a new cluster, switching the Kubernetes Service to it once it is ready, and removing the old one, giving zero-downtime upgrades. On virtual machines, the docs recommend the same shape by hand for significant code changes: create a new Ray cluster, deploy there, and move traffic, instead of mutating the live one.

The rule I give teams is a decision, not a mood. If the change touches the image, the Ray version, the Python version, or a native dependency, it is cluster-level, and you stand up a fresh cluster. If it touches only the Serve config, it is application-level. If you are unsure, it is cluster-level.

The fresh-cluster pattern has a cost you must plan for: for the overlap window you pay for two clusters, and for GPU workloads that can be a large number. Schedule upgrades, tell finance, and keep the overlap short by warming the new cluster with synthetic traffic before the switch.

An upgrade runbook:

1. Read the release notes for every version between your current and target. Serve defaults have changed in minor releases.
2. Build the new image, with dependencies baked in and pinned.
3. In a staging cluster, deploy every tenant's real config, not a sample.
4. Replay recorded traffic and compare latency and error rates with the old cluster.
5. Roll to production with the cluster-level path, watching serve status and your latency dashboards.
6. Keep the old cluster until the rollback window closes, then remove it.

The most important line in that table is the one about defaults. When a default changes between versions, a config that said nothing now says something different, and no error appears. That is how an upgrade that "worked" doubles a latency percentile. Make every tenant config explicit about max_ongoing_requests and target_ongoing_requests, so an upstream default change cannot move your behaviour.

Security: the trust model

Start with an uncomfortable fact: Serve HTTP endpoints have no built-in authentication. A route is reachable by anyone who can reach the proxy. Authentication, authorization, and rate limiting are your job, and they belong in front of Serve or inside the ingress deployment.

There are three places to put them. At the edge, an API gateway or ingress controller authenticates callers and passes identity downstream; this is the right place for most platforms, because it gives one policy surface. Inside the ingress deployment, a FastAPI dependency can validate a token or check a tenant claim, which is useful for per-route rules the gateway cannot express. For internal-only services, network policy in Kubernetes may be all you need, as long as you accept that "inside the cluster" means "trusted".

The second fact is worse. The Ray Dashboard and the jobs API on port 8265 can run arbitrary code on your cluster. Anyone who can reach that port can submit a job. Ray's security documentation says this plainly, and you should read it before you design access. Never expose the dashboard publicly, and treat the --dashboard-host=0.0.0.0 option as a deliberate decision to widen an attack surface, acceptable only inside a network you control. Your CI system needs to reach 8265 to run serve deploy -a; give it a narrow network path and an identity, not an open port, or have the cluster pull config from Git instead of being pushed to.

The trust model, then, has four boundaries:

• Callers to the proxy. Authenticated at the edge. Rate limited per tenant.
• Operators to the dashboard. Network restricted, authenticated, audited. This is the cluster's root.
• Tenant code to the cluster. Tenant code runs inside your cluster as actors. Anything one tenant's code can do, it can do to the others, because they share processes' host, object store, and network. This is not a sandbox.
• The cluster to the outside. Egress policy decides what a compromised replica can reach.

The third boundary is the one that surprises people. A runtime_env isolates Python dependencies, so tenants can use conflicting package versions; it is not a security boundary. Version 2.58 lists experimental sandboxing among its Ray Core changes, but I would not design a trust model around an experimental feature. If tenants do not trust each other, give them separate clusters.

Secrets. Do not put keys in the Serve config or in runtime_env environment variables that live in Git. Mount Vault-delivered or Kubernetes secrets as files or environment variables on the pods, and have the replica read them in __init__. Rotate by redeploying; do not rely on a replica noticing a changed secret.

Supply chain. The handling-dependencies documentation warns against runtime_env pip packages that compile from source at run time, because they burn resources and can destabilise a cluster. The platform answer is stronger than the warning. Build dependencies into the container image, scan the image, pin by digest, and use runtime_env only for small, pure-Python variation. Read the Docker senior guide for the image-hardening checklist; everything there applies to a Ray image. A runtime install on every replica start means your production cluster downloads and executes code from a package index at the worst possible moment, during a scale-up under load. It also means a registry outage can stop you from scaling.

A regional note. For employers in Gulf states and Egypt, data-residency rules often decide where a model may run and where its logs may live. Ray's logs and the dashboard can contain request data if tenants log it, so the platform's logging policy should say what must not be logged, and the cluster's region should be a documented property of each tenant.

Multi-tenancy: what is shared and what is not

Serve supports many applications on one cluster, each with its own name and route prefix. That is a convenience, and it is easy to mistake it for isolation. Applications share the cluster's nodes, the object store, the dashboard, and the controller. A tenant that consumes all memory on a node affects its neighbours; a tenant that crashes a node takes the others on that node down with it.

You therefore have a spectrum of options, and the choice is a business decision more than a technical one.

Model multiplexing deserves its own warning. @serve.multiplexed loads many models per replica with an LRU cache, selected by the serve_multiplexed_model_id request header. It is excellent for a long tail of per-customer models. But the models share a process, and a cache that holds three models per replica means a request for a fourth triggers a load, which is latency and memory churn. Size max_num_models_per_replica from measured model sizes and set the replica's memory to match.

For a shared cluster, the isolation you can actually build comes from a few levers. Resource requests per replica, so a tenant cannot take more than it declared. A quota on max_replicas per tenant, enforced by your config validator. Distinct route prefixes, enforced the same way, so one tenant cannot claim another's path. Separate Kubernetes namespaces and network policy for clusters per class. And a rule that anything handling regulated data lives on its own cluster.

There is also the question of noisy neighbours on GPUs. Fractional GPUs, such as num_gpus: 0.5, let two replicas share a device, but the fraction is a scheduling number and not a memory limit: the framework inside each replica decides how much device memory it takes. Two tenants who each reserve half a GPU and each load a model needing seventy percent of its memory will fail in ways that look random. Fractional GPUs are for models you have measured, owned by one team, not for strangers.

Document the tenancy model in one page that a new tenant reads on day one. It should say what is shared, what a tenant may not do, how quotas work, and what happens when a tenant exceeds them.

Cost: the knobs and the guard rails

Ray Serve cost is mostly accelerator hours, so cost control is mostly utilisation control. The levers, in order of effect:

Batching. @serve.batch converts many small requests into one efficient accelerator call. Its parameters are max_batch_size (default 10), batch_wait_timeout_s (default 0.01), and max_concurrent_batches (default 1). The docs advise powers of two for batch size and a wait timeout below your latency objective minus inference time. The trade is explicit: each millisecond of waiting buys batch fill and costs latency. For a GPU model, batching is usually the single largest saving available.

Right-sized replicas. A replica that reserves four CPUs and uses one wastes three on every copy, multiplied by the replica count. Review ray_actor_options against measured usage on a schedule.

Scale-to-zero for spiky or batch-like tenants. It costs nothing while idle and costs a cold start on the first request. Make it an opt-in with a stated tolerance for that first-request delay.

Fractional GPUs and multiplexing for the cases above.

Ceilings. The cost incident I see most often is not waste, it is runaway. A tenant sets max_replicas: 200, a retry storm arrives, and the node autoscaler happily buys two hundred GPUs. Guard against it with a cluster-level node ceiling that is lower than the sum of tenant maximums, an alert on node count, and a budget alarm in your cloud account. The ceiling should be a configuration the platform owns and tenants cannot edit.

A simple cost model gives you something to show leadership: cost per thousand requests per tenant equals node cost per hour divided by requests served per hour, apportioned by the share of replica-hours each tenant used. It will be an approximation, because replicas share nodes, but approximate and published beats exact and absent. Chargeback changes behaviour faster than any guideline.

Observability designed around objectives

The Beginner view of monitoring is the dashboard and serve status. The platform view starts from objectives: what do we promise, and what measures it?

For each tenant, define an availability objective and a latency objective, measured at the proxy because that is where users experience them. Everything else is diagnosis. The sources you have:

• serve status and serve config, for desired versus actual state. Use them in automation.
• The dashboard, for replicas, nodes, and logs. Human diagnosis, not alerting.
• Logs under /tmp/ray/session_latest/logs/serve/ on each node, under the ray.serve logger. LoggingConfig, available since Ray 2.9, sets JSON encoding, levels, and whether access logs are on. Standardise on JSON and ship the files off the node, because node-local logs vanish with the node.
• The X-Request-ID header, which correlates proxy and replica log lines for one request. Require that tenants propagate it.
• Prometheus metrics. Start Ray with --metrics-export-port to expose them, scrape with Prometheus, and chart in Grafana. Serve metrics are prefixed ray_serve_; list the exact names from your running cluster's metrics endpoint rather than copying them from a blog.
• Tracing, through OpenTelemetry, when you need to see a request cross deployments. Check the current docs for the supported setup in your version.
• Memory profiling with RAY_SERVE_ENABLE_MEMORY_PROFILING=1, for hunting a leaking replica.

Two environment variables are worth knowing as an owner: RAY_SERVE_REQUEST_LATENCY_BUCKETS_MS sets histogram buckets for request latency, which matters because default buckets may not resolve your objective, and RAY_SERVE_METRICS_EXPORT_INTERVAL_MS controls export frequency. Set the first to bracket your tightest objective, or your percentile charts will be coarse exactly where you care.

Alerts, in priority order. First, objective burn: error rate or latency percentile beyond budget at the proxy. Second, control-plane health: the controller or any application not in a running state. Third, capacity: replicas pending for longer than a threshold, node count near the ceiling. Fourth, restarts: a replica restarting repeatedly points to an out-of-memory kill or a failing constructor. Page on the first two; ticket the rest.

Logging deserves a policy because logs are where data-residency problems hide. State what tenants must not log, such as personal data in request bodies, and sample verbose logging rather than enabling it everywhere.

Governance: the platform contract

A platform is a promise plus a process. The promise is what tenants get; the process is how they get it without opening tickets for everything.

The onboarding path should be a template repository. It contains a deployment skeleton with explicit max_ongoing_requests, an autoscaling block, a health check, logging configuration, and a Dockerfile derived from the platform's base image. A new team copies it, changes the model, and opens a pull request. The CI pipeline lints the Serve config against platform rules, builds the image, runs a smoke test that starts the application locally and sends a request, and deploys to staging. Promotion to production is a second approved pull request.

The rules the validator enforces should be few and automatable:

1. Every deployment sets max_ongoing_requests and an autoscaling block with min_replicas and max_replicas.
2. No num_replicas alongside autoscaling_config.
3. max_replicas within the tenant's quota.
4. Declared resources: ray_actor_options present, with memory set for model deployments.
5. No runtime_env pip installs in production configs; dependencies live in the image.
6. Route prefixes within the tenant's namespace.
7. Logging configuration present, and no secrets in the config text.
8. A named owner and an on-call contact in the metadata.

The service tiers are the other half. A gold tier gets a minimum replica count that survives a node loss, a tighter latency objective, and paging. A bronze tier gets scale-to-zero allowed, best-effort capacity, and a ticket queue. Writing the tiers down turns arguments about priority into a choice tenants make with their budget.

Include a deprecation policy: you will tell tenants how long they have to leave a removed field or an old Ray version, and you will enforce it. The upgrade table earlier is the content of that policy. Platforms that never remove anything end up supporting three Ray versions on three clusters.

Finally, own the ownership map. Every application on the cluster should answer: who owns it, who is on call, what is its tier, what data does it touch, where may it run. An application with no answer to those is an orphan, and orphans are what you find at three in the morning.

Incident playbooks

These are written to be read under pressure. Each starts with the symptom, then the first check.

Deployment stuck deploying or unhealthy. Run serve status and read the message. "Waiting for resources" means a scheduling problem: look at node capacity and whether the requested GPU type exists. A constructor exception shows in the replica logs under the Serve log directory. A model that takes longer than the health check allows to load will be killed repeatedly, which looks like a crash loop; compare load time with the health-check timeout, which is configurable per deployment.

Latency climbs, errors do not. Requests are queueing. Check ongoing requests per replica against max_ongoing_requests. If replicas are saturated and the count is at max_replicas, you are capacity-bound: raise the ceiling or shed load. If replicas are not saturated but latency is high, suspect a blocked event loop in a tenant's async handler, and confirm by reading a stack dump of the replica process.

Replicas dying with actor-death errors. Usually memory. Check the node's out-of-memory events and the deployment's memory setting; reduce batch size, raise the reservation, or move the tenant to larger nodes. A replica killed by memory pressure will restart into the same condition if nothing changes, so repeated restarts are the signal to stop and fix.

Scale-up that never arrives. Replica autoscaling asked for more, node autoscaling did not deliver. Look at the autoscaler's view of pending resource requests. The common causes are a quota in the cloud account, an unavailable accelerator type in the region, or a node group at its ceiling.

A tenant melts a shared cluster. Contain first, diagnose second. Reduce that tenant's max_replicas or delete its application with serve.delete, or through the config, which affects only that application. Then analyse. The cost of removing a misbehaving tenant is a degraded tenant; the cost of waiting is every tenant.

The dashboard or jobs port was exposed. Treat it as a full compromise of the cluster. Cut the network path, rotate every secret the cluster could read, rebuild from a known image rather than cleaning the existing nodes, and review what jobs ran. This is the single incident for which in-place repair is the wrong answer.

Bad rollout. The platform's best rollback is the previous Git commit. For application-level changes, redeploy the previous config. For cluster-level changes, switch traffic back to the old cluster, which is why you keep it until the rollback window ends.

Image pull or package index outage during scale-up. Replicas cannot start because the registry or package index is unreachable. Mirror images into a registry you control, keep dependencies in the image, and avoid runtime installs; then this incident class mostly disappears.

For each playbook, rehearse it. A runbook that has never been executed is a hypothesis.

Ray Serve LLM as a tenant class

Ray Serve has an LLM module, ray.serve.llm, that serves models with an OpenAI-compatible interface, built around an LLMConfig and an application builder. It supports vLLM and SGLang as engines and offers multi-model serving, several parallelism modes, prefill-decode disaggregation, and multi-LoRA. In 2.58, KV-cache and token-aware request routing is complete, with tokenization done in-process on the ingress replica and tokens passed on so the engine does not tokenize again. Install it with pip install "ray[llm]", which brings the engine stack and normally needs a GPU.

Treat LLM serving as its own tenant class with its own rules, because its economics differ from a classifier. The cost per replica is a whole GPU or several, the load time is minutes, and the unit of work is a long, streamed generation rather than a quick request. Autoscaling targets for classic request counts fit it poorly; read the Serve LLM documentation for the autoscaling guidance specific to the engine and routing you use, and load-test with realistic prompt and output lengths rather than a fixed small prompt.

Platform concerns specific to this class:

• Admission control. Without authentication and token budgets at the edge, one caller can occupy a GPU for hours. Rate-limit by tokens, not only by requests.
• Warm capacity. A cold LLM replica is minutes away. Keep a minimum, and keep model weights close to nodes so a new replica does not download tens of gigabytes.
• Streaming and timeouts. Streamed responses hold a connection for a long time; make sure every hop, including the load balancer, allows it.
• Version pinning. Engine and Ray versions interact closely. Pin both in the image and test upgrades as cluster-level changes.
• Engine choice. The engines are separate tools with their own guides: vLLM and SGLang. If you only need one model on one engine and no composition, running the engine directly behind your gateway is simpler.

Since the exact LLMConfig fields and builder signatures change across releases, copy the current example from the Serve LLM page for your version instead of from this guide.

Where Ray Serve stops

Every tool has a boundary, and a platform owner should be able to state it without defensiveness.

Ray Serve earns its complexity when you need replica scaling, GPU sharing, composition of several models and business logic in Python, or LLM serving. It is a poor choice when none of that applies. A single small model on one machine is served well by FastAPI alone; adding a Ray cluster adds a controller, a head node, an object store, and an operations burden for no benefit.

Some of these combine rather than compete. Ray Serve can host a Triton or vLLM engine inside a deployment. The question is whether the Python-native composition and unified autoscaling are worth a Ray cluster to your organisation. If your teams are mostly Python data scientists who want to ship composed pipelines, often yes. If your organisation is built around Kubernetes custom resources and does not otherwise run Ray, the extra control plane is a real cost, and you should say so.

There is also a people question. Ray Serve platforms need someone who understands Ray scheduling, not just Serve. If the person who does is leaving, that is a risk as concrete as a technical one.

The review checklist

Use this for design documents and for onboarding a new tenant. Anything that fails is a conversation, not necessarily a block.

Architecture. Which proxy placement is used, and is there a load balancer in front? What is the head-node failure plan? Is the controller monitored?

Capacity. Does the sum of tenant max_replicas fit the cluster, and is the overcommit deliberate? Is there headroom to lose one node? Is there a node ceiling below the sum of maximums?

Request path. Is max_ongoing_requests explicit and above the target by 20 to 50 percent? Are there blocking calls in async handlers? Is there a request timeout?

Config. Is the Serve config in Git, reviewed, and applied by automation? Does the pipeline poll status after deploy? Are old field names gone?

Security. Is there authentication at the edge? Is port 8265 unreachable except from named hosts? Are secrets outside Git? Are dependencies in the image, scanned, and pinned?

Tenancy. Is the isolation level appropriate for the data? Are quotas enforced? Is each fractional-GPU pairing measured?

Cost. Is batching used where it helps? Is scale-to-zero limited to tolerant tenants? Can a tenant see its own spend?

Operations. Are there objectives, and are the alerts tied to them? Are logs shipped off-node in JSON? Is every playbook rehearsed? Is the upgrade path written, with a rollback?

Ownership. Does every application have an owner, an on-call contact, and a tier?

The complete picture

Ray Serve is a thin, Python-native serving layer over a distributed runtime. Its power is that deployments, composition, batching, multiplexing, and autoscaling are all expressed in ordinary code and a small config file. Its cost is that you also run the runtime. As a platform owner, you hold three things together.

First, the failure domains: replicas heal quickly, nodes cost capacity, proxies are redundant, and the controller and head node are the quiet single points whose loss shows up late. Second, the contracts: explicit concurrency and autoscaling settings, quotas, a validator, and tiers, so tenants cannot accidentally reconfigure your platform. Third, the boundaries: the dashboard port is root, tenants share a cluster and not a sandbox, and runtime installs are a supply-chain risk.

Everything else follows from those. Upgrades are fresh clusters when the image changes. Cost is utilisation, held in check by ceilings. Incidents are named, rehearsed, and contained before they are diagnosed. And the honest answer to "should this be Ray Serve?" is sometimes no.

Where the series leaves you

You began by writing a deployment that returned a string. You can now describe the architecture beneath it, predict which layer saturates first, set defaults that protect other teams, design the trust boundaries, run upgrades without downtime, and argue for or against the tool with evidence.

The natural next steps are practical. Build a staging cluster that mirrors production and use it to rehearse an upgrade and an incident. Write your platform's one-page tenancy contract and onboarding template. Read the neighbouring guides: Ray for the runtime beneath Serve, Kubernetes and Helm for the deployment substrate, Prometheus and Grafana for the observability stack, and vLLM if LLM serving is where your platform is heading. Then keep the habit that matters most: when a default, a flag, or a behaviour is in question, check the documentation for the version you actually run.

Sources

• Ray Serve overview
• Ray Serve production guide
• Deploy on Kubernetes with KubeRay
• Deploy on VMs
• Advanced autoscaling
• Dynamic request batching
• Handling dependencies
• Monitoring Ray Serve
• Ray Serve API reference
• Ray Serve LLM
• Installing Ray
• Ray releases on GitHub

Part three of three. At this level an interviewer is not checking whether you know the decorator. They want to see whether you can design a serving platform, run it through an incident, and defend a trade-off to someone who controls the budget. Read the fast review first, then work through each prompt out loud, with a timer.

The thirty-second answer for a senior

Ray Serve is a Python-native model serving library that runs on a Ray cluster. You write deployments, which are classes or functions that become replicas, bind them into applications, and describe the whole thing in a YAML config. As a platform owner, you run the cluster, set the contracts other teams deploy under, and are accountable for upgrades, security, cost, and incidents.

Say the next sentence unprompted, because it separates seniors from the rest: Serve gives you scaling, composition, and batching, but it does not give you tenant isolation or authentication. You build those around it.

The mental model to have ready

Draw this on the whiteboard before you answer any design question, and annotate it as you go. The controller is not in the request path, so a controller outage freezes changes while traffic continues. Proxies are in the path but redundant. Replicas are the cost centre and the unit of recovery. The head node is the part of the cluster you must have a plan for.

Four facts to quote precisely, because precision signals experience:

• max_ongoing_requests defaults to 5 since Ray 2.32, down from 100, and it was named max_concurrent_queries. The docs advise setting it 20 to 50 percent above target_ongoing_requests.
• target_ongoing_requests defaults to 2.0 since 2.32, up from 1.0.
• The scale-down delay defaults to 600 seconds, and the scale-up delay to 30.
• Serve HTTP endpoints have no built-in authentication, and the dashboard on port 8265 can execute arbitrary code.

If you cannot recall a number, say so and say where you would look. A confident "I would check the autoscaling docs for the current default" is better than a wrong figure.

System design: a shared model-serving platform for ten teams

The prompt. Your company has ten data science teams, each with two to five models, mixed CPU and GPU, some with strict latency objectives. Design the serving platform.

A strong answer, in order.

Clarify first. Ask for the latency and availability objectives per team, the traffic shape (steady or spiky), the sensitivity of the data, the regions allowed, and whether teams trust each other. These answers decide the isolation model. Say that out loud: "my tenancy decision depends on data sensitivity, so I will ask."

Choose the substrate. Ray Serve on Kubernetes through KubeRay and its RayService resource, because the docs describe it handling health checking, failure recovery, and upgrades, and because your Git workflow can drive it. Name the alternative you rejected and why: for example KServe if the organisation already standardises on Kubernetes custom resources and the models are mostly single-model services, and say that you would choose Ray Serve here because teams want Python composition and shared GPU scheduling.

Choose the tenancy model. Many applications on a small number of clusters, split by class: one for CPU services, one for GPU services, and a separate cluster for anything regulated. Explain what is shared on one cluster: nodes, the object store, the dashboard, the controller. Then say the sentence the interviewer is listening for: "A runtime environment isolates dependencies, not security, so untrusted tenants get their own cluster."

Define the contract. A template repository with explicit max_ongoing_requests, an autoscaling block, declared resources, health checks, JSON logging, and a Dockerfile from a platform base image. A validator in CI that rejects a config with no max_replicas, with both num_replicas and autoscaling_config, with pip installs in runtime_env, or with a replica ceiling above the tenant's quota.

Deployment flow. Pull request, validation, image build and scan, staging deploy with a smoke test, approved promotion, serve deploy or the operator applying the RayService, then polling serve status because a successful deploy call only means the request was received.

Scaling. Two layers: replica autoscaling on ongoing requests, and node autoscaling underneath. A minimum of one or more for anything user-facing, scale-to-zero by agreement only, and a node ceiling below the sum of tenant maximums. For GPU tenants, a minimum that survives the loss of one node.

Security. Authentication at an API gateway, port 8265 reachable only from named hosts, secrets from a secrets manager mounted at the pod, dependencies baked into scanned images, and egress policy.

Observability. Objectives per tenant measured at the proxy, Prometheus metrics, JSON logs shipped off-node, request IDs propagated, and alerts on objective burn and control-plane health.

Cost. Batching, right-sized replicas, per-tenant replica-hour reporting.

Close with risks. Head-node failure, noisy neighbours, a tenant's blocking async code, and the upgrade overlap cost. Interviewers reward candidates who volunteer the weaknesses of their own design.

What a weak answer does. It describes a single cluster with every team in it, no quotas, no authentication, and serve deploy run by hand from laptops.

Follow-ups to prepare.

How do you stop one team starving another? Declared resources per replica, a replica quota enforced in CI, a node ceiling, and for GPU-heavy teams a separate node pool. Be honest that on a shared cluster these are soft limits.

How do you onboard team eleven in a day? The template and the validator. If it takes a ticket, the platform is the bottleneck.

System design: serving an LLM to the whole company

The prompt. The company wants an internal OpenAI-compatible endpoint backed by open models. Design it on Ray Serve.

A strong answer.

Start with the module. Ray Serve LLM, under ray.serve.llm, exposes OpenAI-compatible chat, completions, and embeddings, and uses vLLM or SGLang as the engine. In 2.58 its request routing is KV-cache and token-aware, with tokenization done once on the ingress replica. Say you would take the current example from the docs for your version because the config fields move between releases.

Then say why LLM serving is a separate tenant class. A replica needs one or more whole GPUs, loading takes minutes, and a request is a long streamed generation. That changes four decisions.

Admission. Authenticate at the gateway and enforce token budgets, not only request counts, otherwise one caller can hold a GPU. Rate limiting by tokens per minute per tenant is the standard shape.

Capacity. Keep a warm minimum. Cold start is the model download plus load. Put weights on fast storage near the nodes or bake them into the image, and say that scale-to-zero is for batch workloads only.

Autoscaling. The default request-count targets were designed for short requests. Load-test with realistic prompt and output lengths and read the LLM docs for the guidance specific to the routing you use.

Streaming. Check every hop allows long-lived streamed connections, including the load balancer, and set timeouts on purpose.

Then discuss when you would not use Ray Serve for this. One model, one engine, no composition: run vLLM behind the gateway directly and avoid running a Ray cluster. Ray Serve earns its place when you need several models, shared autoscaling, or composition of retrieval and generation in one Python application.

System design: zero-downtime upgrade of a production cluster

The prompt. You must move a live Serve cluster from one Ray version to another. Walk me through it.

A strong answer splits the change in two. A change that touches only the Serve config, such as autoscaling parameters, is application-level and applies in place. A change to the image or Ray version is cluster-level. On KubeRay, the docs describe the operator creating a new cluster, switching the Kubernetes Service once it is ready, and then removing the old one. On virtual machines, the docs recommend creating a new cluster, deploying there, and moving traffic.

Then the runbook, which is where seniority shows:

1. Read the release notes for every version in between, looking for changed defaults, removed names, and dependency changes such as Pydantic and Python versions.
2. Build the new image with pinned dependencies and scan it.
3. Deploy every tenant's real config to staging and replay recorded traffic, comparing latency percentiles and error rates against the old cluster.
4. Warm the new cluster before the switch, because model load is the slow part.
5. Switch traffic, watch objective dashboards and serve status.
6. Hold the old cluster until the rollback window closes.

Then the cost and the risk. You pay for two clusters during the overlap, which for GPUs is significant, so you schedule it and tell finance. The silent risk is a changed default: since 2.32 the default max_ongoing_requests and target_ongoing_requests changed, and a config that never stated them changes behaviour without an error. Your mitigation is to make every config explicit, and to diff behaviour in staging.

Incident response

For each question below, structure the answer the same way: contain, diagnose, fix, prevent. Interviewers listen for the order.

Latency has tripled but the error rate is flat

Say what flat errors imply: requests are being served, slowly, so they are queueing or running slowly. First check ongoing requests per replica against max_ongoing_requests and the replica count against max_replicas. If replicas are saturated and at the ceiling, you are capacity-bound; the containment is to raise the ceiling if the cluster has room, or shed lower-tier load. If replicas are not saturated, suspect something inside the replicas: a blocked event loop from synchronous work in an async def handler, a model that got slower, a downstream dependency. Confirm with a stack dump or the replica logs. Then the fix, and the prevention: a linter rule against blocking calls in async handlers and a latency objective alert that fires before users complain.

Deployments are stuck in a deploying state

Read serve status first; it names the problem. "Waiting for resources" means scheduling: is the requested GPU type available, is the node pool at its ceiling, is a cloud quota exhausted? A constructor error shows in the replica log. A model that loads slower than the health check allows will be killed and restarted repeatedly, which looks like a crash loop; compare load time with the health-check timeout. Say that you would check the logs under the Serve log directory on the node, and that an on-node log is gone if the node is, which is why you ship logs off-node.

Replicas keep dying

An actor-death error usually means memory. Check node memory events and the deployment's memory setting. Containment: lower batch size or increase the reservation. The follow-up an interviewer likes: "why does it keep restarting into the same failure?" Because the controller restores the desired replica count, and nothing about the cause has changed. The fix is to change the cause, not the restart.

The controller is down

This is the question that tests whether you know the architecture. Traffic continues for routes that exist; deploys, autoscaling responses, and replica replacement stop. Your concern is drift: if a replica dies while the controller is down, nothing replaces it. The response is to restore the controller, then verify serve status shows every application running and replica counts match the config. Prevention is an alert on controller and application health, separate from traffic alerts, because traffic dashboards stay green.

The dashboard port was reachable from the internet

Treat it as a full compromise: the dashboard and jobs API can execute arbitrary code. Cut the network path first. Then rotate every secret the cluster could read, rebuild from a trusted image rather than cleaning nodes, and review which jobs ran. Prevention is network policy, no public binding, and an external scan in your regular checks. Do not minimise this answer; the point is that you know the severity.

Arguing trade-offs to leadership

Senior interviews often end with a persuasion scenario. The pattern is the same each time: state the decision, state what you are trading, quantify, and propose a reversible step.

"Why not just let each team run its own Ray Serve?"

Concede what is true: independence is faster at first. Then the costs. Ten clusters mean ten upgrade projects, ten security reviews, ten idle GPU pools, and ten different on-call setups. A shared platform amortises those, and the contract (templates, validation, tiers) keeps it from becoming a bottleneck. Propose a measurable pilot: onboard two teams, compare time to production and cost per thousand requests with their current setup.

"Can we scale to zero everywhere to save money?"

Say yes for batch-like or spiky tenants and no for user-facing ones, and explain the mechanism: a cold replica must schedule, import, load weights, and pass its health check, so the first request pays the model load time. Quantify with the tenant's real load time. Offer the middle path: a minimum of one for gold-tier services, scale-to-zero for bronze, and a published tolerance for the first-request delay.

"Why do we need separate clusters for the finance models? Applications are already separate."

Applications on a cluster share nodes, the object store, the controller, and the dashboard. A runtime environment isolates Python dependencies, not access. If a tenant's code is compromised, it runs inside your cluster as actors. Present it as risk reduction priced against a cluster's running cost, and let the data owner decide the risk appetite.

"Is Ray Serve the right tool?"

Where it fits: Python-native composition, GPU sharing, unified autoscaling, LLM serving. Where it does not: one small model, where FastAPI in a container suffices; strict hard isolation between tenants; organisations built around Kubernetes custom resources that do not otherwise run Ray, where KServe may fit better; and throughput-focused multi-framework inference, where Triton is purpose-built. Stating the boundaries of your own tool is a credibility signal.

What strong answers cover that weak ones miss

Common interview questions with model answers

What is the difference between replica autoscaling and node autoscaling? Replica autoscaling is Serve's controller changing the number of replicas based on ongoing requests per replica, within min_replicas and max_replicas. Node autoscaling is the Ray or KubeRay autoscaler adding machines when replicas need resources the cluster lacks. The first reacts in tens of seconds; the second, for GPUs, takes minutes. The slower one sets your real scale-up time.

Why is max_ongoing_requests set above the target? The target is what the autoscaler aims for; the maximum is what a replica will hold. If they are equal, a replica fills at the moment scale-up is first needed. The docs advise 20 to 50 percent above.

What does a blocked event loop look like? Replicas that pass health checks at idle and then time out together under load, because a blocking call in an async def stalls every request on that replica. Fix by using async libraries, making the method sync so it runs in a thread, or offloading the work.

What does serve deploy returning success tell you? Only that the request was received. The rollout is asynchronous, so automation must poll serve status.

Why avoid pip installs in runtime_env in production? They run at replica start, which is exactly when you are scaling under load; they depend on a package index being available; packages that compile from source burn resources and can destabilise a cluster, as the docs warn. Bake dependencies into a scanned, pinned image.

Final self-test

Answer each in under a minute, aloud. If you stall, return to the section named.

1. Name the five components that can fail and say what users see for each. (Architecture)
2. State the 2.32 changes to two defaults and why config should be explicit. (Defaults)
3. Where do you put authentication, and why is port 8265 special? (Security)
4. What is the difference between an application-level and cluster-level change? (Upgrades)
5. Which layer breaks first for a GPU tenant, and how do you size for it? (Scaling)
6. Why is runtime_env not a security boundary? (Tenancy)
7. Give three controls that stop a runaway bill. (Cost)
8. Describe an LLM tenant's four special requirements. (LLM class)
9. When would you recommend against Ray Serve? (Boundaries)
10. Latency up, errors flat: your first three checks. (Incidents)

If you can answer all ten without notes, and you can draw the platform from memory, you are ready for the senior conversation.

Part three of three. At this level the problems are operational: a cluster that carries ten teams, an upgrade that changed a default nobody wrote down, a port that was open for a month. Start with the error table, then the platform habits, the review checklists, and the playbook cards.

Production traps that appear at scale

Cumulative: everything from Beginner and Mid still applies. These traps cost money or cause incidents rather than failed starts.

Most rows in that table have a common root. Either a default was assumed instead of stated, or a shared resource was assumed to be isolated. If you remember only those two sentences, you will prevent most of what follows.

Traps in depth

The default that moved

Since Ray 2.32, max_ongoing_requests defaults to 5 (it was 100 under the name max_concurrent_queries), and target_ongoing_requests defaults to 2.0 (it was 1.0 under a longer name). A tenant config that said nothing about either value inherited the old defaults on an old cluster and the new ones after an upgrade. Nothing errors. The behaviour changes: a deployment that quietly accepted a hundred concurrent requests per replica now accepts five, and the rest queue upstream.

Fix. Treat explicit settings as part of the platform contract. Your CI validator rejects any deployment that omits them. After an upgrade, diff behaviour in staging with replayed traffic rather than trusting that "nothing changed".

The blocked event loop

An async def handler runs on the replica's event loop. A synchronous call inside it, such as a blocking HTTP client or a heavy NumPy computation, stops every other request that replica is holding, including health checks. It looks like a healthy replica that serves nothing. A sync handler is the safer default for blocking code because it runs in a thread pool.

import asyncio
import httpx
from ray import serve

@serve.deployment(max_ongoing_requests=10)
class Enricher:
    def __init__(self):
        self.client = httpx.AsyncClient(timeout=5.0)

    async def __call__(self, request):
        r = await self.client.get("https://example.internal/feature")   # async client: yields
        return r.json()

Fix. A review rule and, better, a lint rule: no blocking I/O in async handlers. Include a load test in the template that sends concurrent requests, so the failure shows up before production.

Fractional GPU confusion

Setting num_gpus: 0.5 tells the scheduler that two replicas may share a device. It does not cap device memory. Each replica's framework takes what it takes. Two models that each need seventy percent of a card will both be scheduled, and one will fail out of memory, sometimes only under load.

Fix. Pair models only after measuring peak device memory under realistic batch sizes, keep pairs within one team, and record the pairing in the config comments.

The runtime install

runtime_env with pip packages is convenient for a notebook and hazardous for production. It runs on replica start, which means during scale-up, which means when load is already high. It depends on a package index being reachable. The documentation warns specifically about packages that compile from source at run time because they burn resources and can destabilise a cluster. And it lets two nodes end up with different dependency sets.

Fix. Dependencies go in the image, pinned and scanned. runtime_env is for small, pure-Python variation or for working_dir when you have no better route, and your validator rejects pip lists in production configs. See the Docker senior guide for the image checklist.

Silent control-plane loss

The controller is not in the request path. When it dies, traffic continues for existing routes and everything looks fine. Deploys stall, scale-up requests are not acted on, and a dead replica is not replaced. The danger is drift: the cluster is running a smaller or older state than the config says.

Fix. Alerts on application status and controller health that are independent of traffic metrics, and a daily reconciliation job that compares serve config with Git.

Platform-ownership habits

Make the contract executable

A document nobody enforces is a wish. Turn every platform rule into a check in the pipeline. The rules that earn their place are the ones that prevented a past incident: explicit concurrency and autoscaling, a replica quota, declared resources, no pip installs in production, a route prefix within the tenant's namespace, a named owner. Start with five and add one after each incident.

Give tenants a template, not a lecture

New teams copy what exists. Make what exists correct. A template repository with a deployment skeleton, an autoscaling block, a health check, JSON logging, and a platform base image converts best practice into the path of least effort. When you improve the template, open pull requests against tenants rather than emailing them.

Keep the Serve config in Git and out of laptops

Nobody runs serve deploy from a laptop against production. The config is reviewed, a pipeline applies it, and the pipeline polls serve status before reporting success. On Kubernetes, the RayService manifest carries the Serve config in its serveConfigV2 field and the operator applies it; pair it with Argo CD so the Git state is the cluster state. If your pipeline needs to reach port 8265 to deploy, give it a narrow network path and an identity, not an open port.

Pin everything and say so

Pin the Ray version, Python version, base image digest, and every library. Write them in one place. When a pin moves, it moves in a pull request that links to the release notes you read. Read the notes of every version between where you are and where you are going, because defaults change in minor releases.

Treat 8265 as root

The dashboard and jobs API can run arbitrary code on the cluster. Nobody exposes it publicly; the question is who inside the network can reach it and with what identity. Log access to it, and put it on the list of things your regular external scan checks.

Know your recovery time per model

For every tenant, record the time from replica start to healthy. It is your real recovery time after a crash and your real scale-up time on an existing node. Keep it in the ownership map beside the tenant's tier. When someone asks "how fast can this scale?", you have a number.

Cap what the autoscaler can spend

Node autoscaling does what you tell it. Set a cluster ceiling lower than the sum of tenant max_replicas, alert on node count approaching it, and keep a cloud budget alarm as the last line. Decide on purpose whether the cluster is overcommitted.

Publish tenant cost

A per-application view of replica-hours and accelerator-hours, visible to the owning team, does more for efficiency than any policy. Approximate allocation is acceptable; absence is not.

Rehearse

Run a game day each quarter. Kill a node. Kill the controller. Fill a tenant's queue. Rotate a secret. Restore from the previous config. The first rehearsal always finds something; the fifth finds nothing, which is the goal.

Write the deprecation policy before you need it

Tell tenants how long they have to leave an old Ray version or a removed config name, and keep the promise. Platforms that never remove anything end up running three versions on three clusters. The names to scan for in inherited configs: max_concurrent_queries, target_num_ongoing_requests_per_replica, upscale_smoothing_factor, downscale_smoothing_factor, and Pydantic v1 models.

Review checklist for a pull request touching a Serve config

Use it on every tenant pull request. It takes five minutes.

• Every deployment sets max_ongoing_requests explicitly, above the autoscaling target by roughly 20 to 50 percent.
• autoscaling_config has both min_replicas and max_replicas, and max_replicas fits the tenant quota.
• There is no num_replicas alongside autoscaling_config.
• ray_actor_options declares CPU, GPU, and memory that match measured use.
• No pip installs in runtime_env; dependencies are in the pinned image.
• No secret appears in the config, and no secret is passed as a committed environment variable.
• The route prefix is inside the tenant's namespace.
• Logging is configured as JSON, with a note on what must not be logged.
• Handlers do not make blocking calls inside async def.
• Any @serve.batch method is async, takes a list, and returns a list of equal length.
• Any fractional GPU use cites a memory measurement.
• No deprecated field names appear.
• The owner, on-call contact, and tier are named.

Review checklist for a design document

Use it when a team proposes a new service or a new tenant class.

• Objectives for availability and latency are stated and measured at the proxy.
• The load shape is described: steady, spiky, batch, or streaming.
• The cold-start time is measured, and the minimum replica choice is justified against it.
• Capacity: the node type exists in the region, the quota is confirmed, and headroom survives losing one node.
• The tenancy level matches the data sensitivity, and shared-cluster limits are acknowledged.
• Authentication and rate limiting are at the edge, and token-based for LLM services.
• The upgrade path is stated, with the cost of the overlap and a rollback.
• Failure modes are listed with their user-visible effect.
• Cost per thousand requests is estimated, with a ceiling and an alert.
• Data residency: the region is named, and logs stay inside it.
• The alternative considered is named, with the reason Ray Serve was chosen.

Review checklist for an upgrade

• Release notes read for every version between current and target.
• Changed defaults identified and made explicit in tenant configs.
• New image built, scanned, and pinned by digest.
• Every tenant's real config deployed to staging.
• Recorded traffic replayed; latency percentiles and error rates compared against production.
• Cluster-level change planned as a new cluster, with the overlap cost approved.
• New cluster warmed before switching.
• Rollback is switching traffic back, and the old cluster is retained until the window closes.
• Tenants notified with dates and a list of config names to change.

Practice cards

Each card is a short exercise. Do them on a staging cluster, and write down what you found.

Card one: find the knee

Deploy a model with one replica. Send increasing concurrency and record latency at each level. Plot latency against concurrency, find where it bends, and set target_ongoing_requests below that point and max_ongoing_requests 20 to 50 percent above the target. Repeat with @serve.batch at a batch size of 8 and compare the knee.

Card two: break the event loop

Write a deployment with a time.sleep in an async def handler and a sibling with await asyncio.sleep. Send concurrent requests to each and compare the latency. Then watch what happens to a health check during the blocked period.

Card three: kill the controller

On a local or staging cluster, find the controller actor and stop it. Confirm traffic still flows, then kill a replica and note that it is not replaced until the controller returns. Write the alert that would have told you.

Card four: prove the config is explicit

Write a script that reads a Serve config and exits nonzero if any deployment lacks max_ongoing_requests, lacks max_replicas, combines num_replicas with autoscaling_config, or lists pip packages in runtime_env. Run it against three real configs and see how many fail.

Card five: measure recovery

Time how long a replica takes from start to healthy for each tenant model. Put the number in the ownership map. Compare with each tenant's latency objective and decide whether the minimum replica count is safe.

Card six: cap the spend

Set a tenant's max_replicas far above the cluster capacity, generate load, and watch what the node autoscaler tries to do. Then add a ceiling and repeat. Write the alert for the node count.

Card seven: rehearse a rollback

Deploy a change, then roll back by redeploying the previous config. Time it. Do the same for a cluster-level change by switching traffic to the previous cluster. The difference between those two numbers is your rollback cost.

Operational recipes

Check status after every deploy

serve deploy serve_config.yaml -a http://<head>:8265
serve status -a http://<head>:8265

Run the second command in a loop in your pipeline until every application reports running, and fail the job on a failed state or a timeout. The deploy command reports that the request was received; it does not report that the rollout worked.

Compare the running config to Git

serve config -a http://<head>:8265 > running.yaml
diff running.yaml serve_config.yaml

Run it daily. Any difference means someone changed the cluster outside the pipeline, or the controller has not converged. Both deserve attention.

Find the logs on a node

ls /tmp/ray/session_latest/logs/serve/

Controller, proxy, and replica logs live there, and they disappear with the node. If your platform does not ship them elsewhere, a node loss destroys the evidence for the incident that caused it.

Expose metrics

Start Ray with a metrics export port, then scrape that port with Prometheus:

ray start --head --metrics-export-port=8080
curl -s http://localhost:8080 | grep ray_serve_ | head

List the series your version exposes before you write a dashboard, and build Grafana panels from what you see.

Verify the install in the image

python -c "import ray; from ray import serve; print(ray.__version__)"
pip index versions ray

The first confirms what the image contains. The second shows what is available, so you know how far behind you are.

Restrict the dashboard

Start the head node with the default dashboard binding on a network you do not control, and put access through a bastion or an authenticated proxy if people need the UI. Reserve --dashboard-host=0.0.0.0 for networks that are already private and policy-controlled, and check from outside the network that port 8265 does not answer.

The one-page habit list

Print this and put it beside the runbook.

1. State every default that matters; do not inherit it.
2. Alert on the control plane, not only on traffic.
3. Dependencies in the image, pinned and scanned.
4. Port 8265 is root.
5. A shared cluster is not an isolated cluster.
6. Cap what the autoscaler can spend.
7. Know the model load time of every tenant.
8. Config in Git; pipeline polls status.
9. Upgrades change defaults; read the notes and replay traffic.
10. Rehearse the incidents you wrote playbooks for.

When something goes wrong at scale, ask which of the ten was skipped. It is usually one of them.