The Complete Vertex AI Guide

This is part one of three. It covers everything you need to do real work with Vertex AI: set up a Google Cloud project safely, train a model on managed hardware, register it, put it behind an endpoint, ask it for predictions, run a batch job, call a Gemini model, and clean up so you are not billed for things you forgot. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. Cloud platforms are learned by clicking, failing, reading the error, and trying again, and every error message in this guide is one you will probably meet.

What Vertex AI is, and the problem it solves

Vertex AI is Google Cloud's managed platform for machine learning. It gives you hardware to train models on, a place to keep the models you produced, a way to serve them over HTTPS, a way to orchestrate multi-step workflows, and access to Google's Gemini family of generative models. You use it through a web console, a command-line tool called gcloud, a Python SDK, or plain REST calls. They all drive the same underlying service.

To see why it exists, picture what a team does without a platform. A data scientist trains a model on a laptop and saves a model.pkl file. Someone must now rent a server, install the right Python version and library versions, write a small web server around the model, open a port, add authentication, make it restart when it crashes, add more servers when traffic grows, and keep track of which file is the current model. Each of those steps is a small engineering project, and each is done differently on every team. Then the data scientist wants to retrain on a GPU for an hour, and someone has to rent one, install drivers, and remember to switch it off.

Vertex AI takes over the repetitive plumbing. You hand it your training script and say "run this on one machine of this size". You hand it a trained model and say "serve this, between one and three replicas". Google provisions the machines, runs your code, collects the logs, and bills you per unit of compute used. You still decide what the model is and whether it is any good. The platform decides where it runs.

The diagram is the spine of the platform. Almost every feature you will meet later, pipelines, experiments, monitoring, feature stores, is a way of automating, recording, or watching one of those four stages.

What people use it for:

Who it is not for is worth saying early. If you want a model that already exists to answer questions, you may not need to train anything at all, and the generative section near the end of this guide is your shortest path. If you have a notebook experiment and no intention of serving it, the managed notebook feature may be all you need. And if your employer is on another cloud, you will meet the same ideas under other names; the SageMaker guide covers the Amazon equivalent, and nearly every concept transfers.

The mental model: five nouns

Vertex AI has hundreds of pages in its documentation, but a beginner needs only five ideas. Learn them properly and the rest of the platform becomes variations.

Project. Every Google Cloud resource lives inside a project. A project is the unit that holds your billing account link, your enabled services, your permissions, and your quotas. When you delete a project, everything inside it is deleted, which makes projects a convenient sandbox. Every project has a human-readable name, a globally unique project ID such as mlops-lab-123456, and a numeric project number. Code and commands nearly always want the ID.

Location, also called region. Most Vertex AI resources are regional: a model uploaded to us-central1 exists only there, and an endpoint in europe-west4 serves from there. The region is where your data is processed and stored, which matters for latency, for price, and for legal reasons. Not every feature exists in every region, and the official locations page is the authority for what is available where. This is the single most common source of "it worked yesterday in a different region" confusion, so get in the habit of treating the region as part of a resource's name.

Training job. A training job is a request that says "run this code on these machines". The platform starts the machines, runs your code, streams its logs, and shuts the machines down. There are two broad styles. AutoML needs no code: you supply data and the platform picks the model. Custom training runs code you wrote, either in a container Google provides for popular frameworks or in a container you build yourself. This guide teaches custom training because it is the one you will use at work.

Model. After training, you register the result as a Model in the Model Registry. A Model is not a file; it is a record that points to your saved artifact in Cloud Storage and to a serving container, the program that knows how to load that artifact and answer requests. A model can have versions, and versions can carry aliases such as default or champion. Think of the registry as a labelled shelf.

Endpoint. An endpoint is a stable address that serves predictions. You deploy a model to an endpoint, which means placing it on one or more machine replicas that sit behind the endpoint, and you can split traffic between several deployed models. An endpoint with a model on it is running machines that bill by the hour, whether or not anyone is sending traffic. Hold on to that sentence; it is the most expensive thing in this guide.

Two supporting nouns will appear constantly. Cloud Storage (GCS) is Google's object store, and its containers are called buckets, addressed with gs://bucket-name/path. Vertex AI uses a bucket as a staging area for your code and as the place training jobs write their outputs. Service accounts are non-human identities. When a training job runs, it runs as a service account, and the job can touch only what that account is allowed to touch. Later sections show where this bites.

Getting set up: account, project, billing, API

Vertex AI is a service, not a program, so "installing" it means preparing your Google Cloud account. Doing the setup carefully once saves an afternoon of confusing permission errors.

Create or choose a project. In the console, open the project selector and create a new project for this course. Use a fresh project rather than an existing work project: if something goes wrong or you forget to clean up, you can delete the whole project and be certain nothing is still billing.

Link a billing account. Vertex AI is a paid service. New Google Cloud accounts have commonly received free trial credits, but the offer changes, so read the current terms on the sign-up page rather than assuming. Billing must be linked to the project before the API will work. Whatever credits you have, set a budget alert in the Billing section so an email arrives when spend crosses an amount you pick. A budget alert does not stop spending by itself, but it is your early warning.

Enable the API. Google Cloud services are off by default in a new project. Vertex AI's is called aiplatform.googleapis.com. You can enable it from the console by searching for Vertex AI and pressing the enable button, or from a terminal after you install the command-line tool in the next section.

Authenticate. There are two separate kinds of login, and mixing them up is the classic beginner stumble. One is for the gcloud command-line tool itself. The other, called Application Default Credentials (ADC), is what the Python SDK and other libraries look for when your code calls Google. You run a command for each, and the next section does so.

Your own user account, when you created the project, is its owner and can do everything. That is convenient for learning and dangerous at work. In a company project you will usually receive a narrower role, such as Vertex AI User (roles/aiplatform.user), which allows you to run jobs and call endpoints but not to administer the project. If you see a 403 permission error, the question to ask is always "which role does the identity running this need?", and we return to that in the errors section.

Installing the tools and checking the setup

You need two things on your machine: the gcloud command-line tool, and the Python SDK. Neither is large.

Install gcloud. The official installation page covers every platform. On macOS with Homebrew, brew install --cask google-cloud-sdk works; on Linux the page offers apt and yum repositories and a tarball; on Windows it offers an installer. Open a new terminal afterwards so gcloud is on your path, then run the initial configuration:

gcloud init
gcloud auth login
gcloud auth application-default login
gcloud config set project PROJECT_ID
gcloud services enable aiplatform.googleapis.com

Read what each line does, because they are not interchangeable. gcloud init walks you through choosing an account and a default project. gcloud auth login authorizes the command-line tool. gcloud auth application-default login creates the Application Default Credentials that your Python code will find automatically. gcloud config set project sets which project later commands act on; replace PROJECT_ID with the ID, not the display name. The last line enables the Vertex AI API if you have not already.

Install the Python SDK. The current SDK needs Python 3.10 or newer; Python 3.9 and older are deprecated. Always use a virtual environment so the SDK's many dependencies stay out of your system Python.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install google-cloud-aiplatform google-genai

google-cloud-aiplatform is the classic machine-learning SDK: it holds CustomTrainingJob, Model, Endpoint and the other nouns from the previous section. google-genai is the Google Gen AI SDK, used for Gemini and other generative models. They are separate packages on purpose, and the reason matters, so read the next callout.

Check the setup. Run these in order. Each checks a different layer, so when one fails you know where to look.

gcloud config list
gcloud auth list
gcloud ai models list --region=us-central1

The first shows your active project. The second shows which account is active. The third calls Vertex AI itself; on a new project it should print Listed 0 items. and not an error. Then check the SDK from Python:

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1")
print(aiplatform.Model.list())

An empty list [] means everything works. If you see DefaultCredentialsError, you skipped gcloud auth application-default login. If you see a 403 that mentions the API being disabled, run the enable command from the setup section.

Cloud Storage: the bucket every job needs

Before the first training job, make a bucket. Vertex AI uses Cloud Storage as the shared disk between your laptop and its machines: you upload data there, the platform stages your code there, and trained model files are written back there.

Bucket names are globally unique across all of Google Cloud, so choose something with your project in it. Create the bucket in the same region as your Vertex AI resources. A bucket in a different location from your job usually works, but it adds latency and egress charges, and some operations complain.

export REGION=us-central1
export BUCKET=gs://PROJECT_ID-vertex-lab
gcloud storage buckets create $BUCKET --location=$REGION
gcloud storage ls

Upload a small file to prove access, then read it back:

echo "hello" > hello.txt
gcloud storage cp hello.txt $BUCKET/test/hello.txt
gcloud storage ls $BUCKET/test/

The gs:// prefix is how every Vertex AI argument refers to storage. Folders in a bucket are a convention, since object stores have just names with slashes in them, but the console and ls display them as folders. You will pass these paths as the staging bucket, as training data locations, and as model artifact locations.

For the data itself, Vertex AI can also read straight from BigQuery, Google's warehouse, which is often where the company's tabular data already lives. For this guide we will generate data inside the training script so that nothing depends on a dataset you do not have.

Your first training job

This is the section the rest of the guide leans on. We will train a very small model, a scikit-learn classifier, on managed hardware. The model is deliberately boring. What matters is that you see the whole shape of a Vertex AI job: a script, a container, a machine, and an artifact.

Step 1. The training script

Create train.py. It builds a toy dataset, trains a classifier, evaluates it, and saves the model to a location Vertex AI gives it through an environment variable.

import os
import joblib
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=3, n_informative=3,
                           n_redundant=0, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)

model = LogisticRegression().fit(X_train, y_train)
print("accuracy:", model.score(X_test, y_test))

model_dir = os.environ.get("AIP_MODEL_DIR", "./model")
if model_dir.startswith("gs://"):
    local_dir = "/tmp/model"
else:
    local_dir = model_dir
os.makedirs(local_dir, exist_ok=True)
joblib.dump(model, os.path.join(local_dir, "model.joblib"))
print("saved to", local_dir)

Two details deserve explanation. The script prints the accuracy because anything your script prints to standard output becomes a log line you can read in the console, and a training script that prints nothing is a training script you cannot debug. And it saves to a directory taken from AIP_MODEL_DIR, an environment variable that Vertex AI sets for jobs created through the SDK's training-job classes. The platform decides where outputs go and tells your code, rather than your code hardcoding a path.

The exact way the saved file reaches Cloud Storage varies with the container and with how you submit the job. The simplest reliable approach for a beginner is the SDK's CustomTrainingJob, which handles that copy for you when you use a prebuilt training container, so we use it below. If you instead write your own container, you are responsible for uploading the model file to the storage location the job gives you, and the Mid-level guide covers it.

Run the script locally first. This catches every bug that has nothing to do with the cloud, and it is free.

pip install scikit-learn joblib numpy
python train.py

You should see an accuracy line and a "saved to" line. If this fails, do not submit it to the cloud; the cloud will only fail slower and charge you for it.

Step 2. Submit it to Vertex AI

from google.cloud import aiplatform

PROJECT = "PROJECT_ID"
REGION = "us-central1"
BUCKET = "gs://PROJECT_ID-vertex-lab"

aiplatform.init(project=PROJECT, location=REGION, staging_bucket=BUCKET)

job = aiplatform.CustomTrainingJob(
    display_name="train-sklearn-toy",
    script_path="train.py",
    container_uri="us-docker.pkg.dev/vertex-ai/training/sklearn-cpu.1-0:latest",
    requirements=["joblib"],
    model_serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest",
)

model = job.run(replica_count=1, machine_type="n1-standard-4")
print(model.resource_name)

Read it top to bottom. aiplatform.init sets the defaults for every later call: which project, which region, and the staging bucket where the SDK uploads your script. If you leave out the staging bucket, the SDK will refuse to create the job, because it has nowhere to put your code.

CustomTrainingJob describes a job whose code is a local Python script. display_name is the label you will see in the console. script_path is the file to run. container_uri names the container image the script runs inside: Google publishes prebuilt training images for scikit-learn, TensorFlow, PyTorch and XGBoost, with the framework already installed, so you do not build your own. requirements lists extra pip packages to install at start-up. model_serving_container_image_uri names a prebuilt prediction image, the program that will later load your model for serving. Training and serving images are different images with different jobs.

job.run(...) is where the cloud work happens. replica_count=1 asks for a single machine, and machine_type="n1-standard-4" picks four virtual CPUs and a standard amount of memory. By default the call blocks: your terminal waits while Vertex AI provisions the machine, pulls the image, runs the script, and finishes. That takes several minutes even for a tiny job, because starting a machine is not instant. Expect machine start-up time to dominate a job this small.

When it finishes, run returns a Model object, because we gave the job a serving container. The platform took the files your script saved, registered them as a model, and attached the serving image. model.resource_name prints something like projects/123456789/locations/us-central1/models/987654321. That long string is the model's identity, and it shows the region is part of the name.

Step 3. Watch it in the console

While job.run blocks, open the console, go to the training section, and find the custom job. You will see its state move through pending, running, and succeeded, and a logs link that opens Cloud Logging filtered to this job. Open the logs. You should see your accuracy: line among the platform's own messages. This is how you debug a failing job, so make friends with it now.

The Model Registry: versions and aliases

When the job finished, a model appeared. Open the Model Registry page in the console and find it. This is the shelf we mentioned earlier, and it earns its place the moment you train a second time.

A registered model has a name, a region, a pointer to the artifact in Cloud Storage, the serving container to use, and labels you can add, which are key-value tags used for search and for cost reports. The registry also records lineage: which training job produced the model and from what. When someone asks in six months which job produced the model now in production, the registry is where you look.

Versions solve the "retrained, but did it get better?" problem. Rather than creating an unrelated model each time, you upload the new artifact as a new version of the same model. Versions are addressed as MODEL_ID@VERSION. Aliases are movable labels on versions: default points to the version served when nobody says which one, and teams add their own such as champion or candidate. Promoting a model then means moving an alias rather than copying files. The Mid-level guide shows promotion in CI.

From Python you can look at what you have:

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1")

for m in aiplatform.Model.list():
    print(m.display_name, m.resource_name)

And the same thing from the terminal:

gcloud ai models list --region=us-central1

You can also register a model you trained somewhere else, such as on your laptop, without any Vertex AI training job. You upload the artifact to a bucket and call Model.upload, which we use in the next section's alternative. This matters because many real teams train in a notebook and use Vertex AI only for serving.

Deploying to an endpoint and asking for predictions

A registered model does nothing until you deploy it. Deploying creates, or reuses, an endpoint and puts the model on machines behind it.

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1")

model = aiplatform.Model.list(filter='display_name="train-sklearn-toy"')[0]

endpoint = model.deploy(
    machine_type="n1-standard-2",
    min_replica_count=1,
    max_replica_count=1,
)
print(endpoint.resource_name)

model.deploy creates an endpoint for you when you do not pass an existing one. It then provisions a machine of the type you asked for and loads your model onto it using the serving container. Expect this to take several minutes, and it blocks while it works. We set both replica counts to one for a lab: min is the number of replicas that are always running, and max is how far autoscaling may grow under load. For anything real you would set min to at least two for availability, but in a lab one is enough.

Asking for a prediction

Once deployed, call the endpoint with a list of instances. For our model, an instance is a row of three numbers, because the training data had three features:

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1")

endpoint = aiplatform.Endpoint.list(filter='display_name="train-sklearn-toy-endpoint"')[0]
response = endpoint.predict(instances=[[0.5, -1.2, 0.3], [1.1, 0.4, -0.7]])
print(response.predictions)

The result is a list with one entry per instance, here the predicted class, such as [1, 0]. The format of instances and predictions depends on the serving container: the scikit-learn image expects a list of feature rows, and the TensorFlow and PyTorch images have their own conventions. If you send the wrong shape, the error comes back as a 400 with a message from the model server, and reading that message is how you learn the expected format.

Note the filter on display_name. When we let deploy create the endpoint, its name was generated for us; in your own code you will usually create the endpoint explicitly so that you control the name, as in the next block. Use whichever you like, but know which one you used when you look for it later.

endpoint = aiplatform.Endpoint.create(display_name="train-sklearn-toy-endpoint")
model.deploy(endpoint=endpoint, machine_type="n1-standard-2",
             min_replica_count=1, max_replica_count=1)

The terminal has equivalents for the same operations, which are valuable in scripts and in debugging:

gcloud ai endpoints list --region=us-central1
gcloud ai endpoints create --region=us-central1 --display-name=my-ep
gcloud ai endpoints deploy-model ENDPOINT_ID --region=us-central1 \
  --model=MODEL_ID --display-name=DEPLOYMENT_NAME \
  --machine-type=n1-standard-4 --min-replica-count=1 --max-replica-count=10

Endpoints are also reachable over REST, which is what other services will call in production. The URL has the form https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/REGION/endpoints/ENDPOINT_ID:predict, with a bearer token in the Authorization header. The SDK builds that request for you; knowing the shape helps when you debug from curl.

Tearing it down

This is not optional. Deleting an endpoint that still has a model deployed fails, so undeploy first:

endpoint.undeploy_all()
endpoint.delete()

Verify with gcloud ai endpoints list --region=us-central1, which should show nothing. The registered model costs almost nothing to keep, since it is just a pointer and a file in a bucket; the running replicas are what cost money.

Batch prediction: when you do not need an endpoint

Many problems do not need a live service. If you score a million customers every night, or label a folder of images once, an always-on endpoint is wasteful. Batch prediction runs a one-off job: it reads inputs from Cloud Storage or BigQuery, runs your model on temporary machines, writes the outputs, and disappears. You pay only while it runs.

The input for our scikit-learn model is a file of JSON lines, one instance per line, in a bucket. Make one:

cat > batch_in.jsonl <<'EOF'
[0.5, -1.2, 0.3]
[1.1, 0.4, -0.7]
[0.0, 0.0, 0.0]
EOF
gcloud storage cp batch_in.jsonl gs://PROJECT_ID-vertex-lab/batch/in.jsonl

Then submit the job:

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1")
model = aiplatform.Model.list(filter='display_name="train-sklearn-toy"')[0]

batch_job = model.batch_predict(
    job_display_name="batch-toy",
    gcs_source="gs://PROJECT_ID-vertex-lab/batch/in.jsonl",
    gcs_destination_prefix="gs://PROJECT_ID-vertex-lab/batch/out/",
    machine_type="n1-standard-4",
)
print(batch_job.state)

The job writes result files under the destination prefix, in a subfolder it creates for the run. List them with gcloud storage ls --recursive gs://PROJECT_ID-vertex-lab/batch/out/ and read one with gcloud storage cat. The format of the input file must match what the serving container expects, which is the same rule as for online prediction.

The decision between the two styles is a trade-off you should be able to explain.

Calling Gemini: generative AI in the same project

Not every task needs a model you trained. Vertex AI also gives you hosted generative models, notably Gemini, which you call over an API without any training or deployment step. You pay per use, measured in tokens (small chunks of text), rather than for machine hours.

Use the Google Gen AI SDK, which you installed as google-genai. This is the current way to call Gemini on Vertex AI; the older vertexai.generative_models classes belong to the pre-2.0 SDK and are on their way out.

from google import genai

MODEL_ID = "REPLACE_WITH_A_CURRENT_MODEL_ID"

client = genai.Client(vertexai=True, project="PROJECT_ID", location="global")
response = client.models.generate_content(
    model=MODEL_ID,
    contents="Explain a Vertex AI endpoint to a new engineer in two sentences.",
)
print(response.text)

The vertexai=True argument is what routes the call to Vertex AI, with your Google Cloud project, billing and permissions, instead of to the separate Gemini Developer API with an API key. The same client can be configured by environment variables instead: GOOGLE_GENAI_USE_VERTEXAI=true, together with variables for the project and location. Check the Gen AI SDK documentation for the exact names before relying on that form.

Notice that the model name is a placeholder. Model identifiers change quickly: Google retires older Gemini versions on a published schedule, and a hardcoded ID in a tutorial can be dead within months. Gemini 1.5 access, for example, was discontinued in September 2025. Open the "model versions" page in the documentation, pick a current stable model, and put its ID in one constant at the top of your file so that updating it is a one-line change. If you see 404 Publisher Model ... was not found or your project does not have access to it, the ID has been retired or is unavailable in your location.

The location for generative calls is often global or one of several supported regions, and not every model is offered in every region. If data residency matters for your employer, check the models and locations pages for the combination you need before building anything on it. For companies in the Gulf or Egypt, this is a real design constraint, not a footnote: some regulated data may not be allowed to leave a particular country, and the answer lives in the location support tables and in your compliance team's rules.

A chat-style interaction, where the model remembers earlier turns, uses a chat object from the same SDK; the Gen AI SDK documentation shows the current pattern. If you want to build agents that call tools, that is a larger topic with its own package, and the Google ADK guide covers the toolkit Google built for it. The Gemini API guide covers the same models through the developer-facing route.

Notebooks and the console: where beginners actually start

Plenty of people meet Vertex AI through a notebook rather than a terminal, and it is a fine way to learn. Vertex AI Workbench gives you a managed JupyterLab instance running on a Google Cloud virtual machine, with gcloud and the SDK already installed and authenticated as the instance's service account. Colab Enterprise gives you notebooks inside the console without managing a machine. Older "managed notebooks" and "user-managed notebooks" products reached end of support in 2025 and 2026, so when a tutorial tells you to create one of those, use a Workbench instance instead.

A Workbench instance is a machine that bills while it is running. Stop it when you finish for the day, and delete it when the course is over. The console shows a stop button on the instance list.

The console is also where you inspect everything this guide created. The sidebar groups the platform by stage. You will find training jobs, the Model Registry, endpoints, batch predictions, pipelines, and Model Garden, a catalogue of Google, open and partner models you can deploy or try. Learn to read the Logs tab on jobs and the Monitoring graphs on endpoints; they answer most "why did this fail" and "is it being used" questions without any code.

Pipelines, experiments tracking, feature stores and model monitoring are the next layer of the platform. They matter in production, and they build on exactly what you have done here: a pipeline runs a training job, registers the model and deploys it automatically. The Mid-level guide covers them, and the Kubeflow guide explains the pipeline format Vertex AI Pipelines runs. If you are weighing it against open-source tracking tools, the MLflow guide is the natural comparison.

Permissions, regions and quotas

Most failures on a cloud platform are not bugs in your code. They are one of three things: you are not allowed, you are in the wrong place, or you ran out. Learn to recognise them.

Not allowed: IAM. Google Cloud's permission system is called IAM. A principal (a user or service account) is granted a role (a bundle of permissions) on a resource. When your call fails with 403 and a message such as Permission 'aiplatform.endpoints.predict' denied, the message names the missing permission. The fix is usually to grant a role that contains it, such as roles/aiplatform.user, to the identity that made the call. Remember that identity might not be you: a training job runs as a service account, and that account needs access to your bucket and, for private images, to Artifact Registry. Granting yourself roles does nothing for the job's account. Prefer narrow roles; the project Owner role works and is a poor habit.

Wrong place: regions. Create your bucket, your jobs, your models and your endpoints in the same region unless you have a reason not to. A model in us-central1 cannot be deployed to an endpoint in europe-west4. A feature may exist in one region and not another. When NotFound appears for something you are sure you created, check the region in your aiplatform.init call and in the --region flag first. For employers in the Middle East, check the locations page for what each Google Cloud region in the Gulf region offers before you design around it; the supported set of features differs by region and changes.

Ran out: quotas. Google limits how much of each resource a project can use, per region. GPUs are the usual surprise: a new project often has little or no GPU quota, and a job that requests one fails with a 429 RESOURCE_EXHAUSTED message naming the quota metric. The fix is to request a quota increase in the console (which can take time), pick another region, or use a smaller machine. Plan for this before a deadline, not on the day.

Cost control: the habits that protect your wallet

Cloud bills are made of a few line items. Knowing which they are makes the bill predictable.

Endpoint replicas bill per node-hour while deployed, idle or not. Training jobs bill per machine-hour while running, and a job that hangs keeps billing until it times out or you cancel it. Notebook instances bill while running. Cloud Storage bills per gigabyte stored per month, which is small for toy data. Generative calls bill per token. Batch predictions bill for the machines while the job runs.

The habits that follow from this are simple.

1. Work in a disposable project and delete the project when done. Deleting a project begins a grace period before everything is purged, but billing for its resources stops.
2. Set a budget alert the day you create the project, with several thresholds.
3. Undeploy the moment you finish. Make endpoint.undeploy_all() a reflex.
4. Prefer batch prediction when nobody is waiting for the result.
5. Start small. A job on n1-standard-4 for a toy model is plenty; a GPU is for models that need one.
6. Use labels on everything you create so the billing report can be filtered by owner or purpose.
7. Check the console daily during the course for endpoints and instances you forgot.

Prices change and differ by region, so use the official pricing page and the pricing calculator for estimates, and do not trust a number you remember from a blog.

Reading the common errors

These are the failures a beginner meets, with what they mean and what to do. The exact wording can vary between SDK versions and services, so match on the meaning and the status word rather than on every character.

DefaultCredentialsError: Your default credentials were not found. Your code asked for Application Default Credentials and none exist. Run gcloud auth application-default login. Remember that gcloud auth login is a different credential and does not satisfy this.

403 ... Vertex AI API has not been used in project ... before or it is disabled. The API is not enabled. Run gcloud services enable aiplatform.googleapis.com, wait a minute, and retry.

403 Permission ... denied. The identity lacks a permission. Read the permission named in the message, find which identity ran the call, and grant a role containing it. Do not guess which identity: for a training job, it is the job's service account.

404 ... not found. Either the resource does not exist, it exists in another region, or, for a Gemini model, the model was retired. Check the region and the exact ID before assuming deletion.

429 RESOURCE_EXHAUSTED or a quota message. You hit a limit. For a GPU, request quota or change region. For generative calls, add retries with backoff, or ask for more quota.

400 ... staging bucket errors. aiplatform.init needs staging_bucket="gs://..." for custom training, and the bucket must exist and be reachable.

FailedPrecondition: Model server never became ready. Deployment started the serving container and it never answered its health check. Open Cloud Logging for the endpoint, read the container's output, and look for a Python exception while loading the model. A mismatch between the library version you trained with and the one in the serving image is a typical cause: a scikit-learn model pickled with one version may not load in an image with another. Train and serve with matching versions.

Image pull failures at job start. The container_uri is wrong or retired, or the job's service account cannot read a private repository. Copy a current URI from the docs, and for private images grant the right Artifact Registry reader role.

ImportError or ModuleNotFoundError for vertexai.generative_models. Your code is from before the 2.0 SDK. Move to google-genai, as in the Gemini section.

A job that "succeeds" but produces nothing. Your script wrote the model somewhere other than where the platform looks. Print the output path in your script, read it in the logs, and check it against the model directory the job reports.

Putting it all together

Here is one small end-to-end project that uses every piece. You will build it from scratch, tear it down, and write down what each step cost you in time and money. Treat it as a rehearsal for a real interview task.

Goal. Train a classifier on managed hardware, register it, serve it for five minutes, score a batch file, call Gemini to explain the result in plain language, and delete everything.

1. Project. Create a new project, link billing, set a budget alert, and enable the Vertex AI API.
2. Tools. Install gcloud, run the login and ADC commands, create a virtual environment, and install google-cloud-aiplatform and google-genai. Pin the versions.
3. Bucket. Create a bucket in your chosen region.
4. Train. Run train.py locally, then submit it with CustomTrainingJob using current container image URIs.
5. Register. Confirm the model in the Model Registry and add a label.
6. Serve. Deploy to an endpoint with one replica, send predictions from Python, then from curl using the REST URL and a token from gcloud auth print-access-token.
7. Batch. Upload a JSON-lines file and run batch_predict.
8. Explain. Call Gemini with a prompt that includes the predictions and asks for a one-paragraph summary for a non-technical manager.
9. Clean up. Undeploy and delete the endpoint, delete the bucket's test objects, and stop any notebook instance. Verify with gcloud ai endpoints list and the console.
10. Write down. In a short README, record the region, the SDK version, the model ID you used, the time each step took, and anything that surprised you.

# the REST call from step 6, after you substitute your own values
curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID:predict \
  -d '{"instances": [[0.5, -1.2, 0.3]]}'

The call returns JSON with a predictions array. If you get 401, your token expired or you are logged into the wrong account; if 404, check the endpoint ID and region in the URL.

What you can now do, and what comes next

If you worked through the Try it tasks, you can do things many people who list Vertex AI on a resume cannot. You can set up a project safely, with a budget alert and no leaked keys. You can authenticate the command-line tool and the SDK and know which login is which. You can train on managed hardware from a script, read the logs of a job, and register the result. You can deploy to an endpoint, call it from Python and from REST, and tear it down so it stops billing. You can run a batch job when an endpoint is overkill. You can call a Gemini model through the current SDK and know that model IDs rotate. And you can read the platform's errors by their code and their sentence rather than by panic.

You also know what you have not yet touched. Pipelines automate the whole sequence you just did by hand and record each run. Experiments track parameters and metrics across runs. Custom containers let you serve anything, not just frameworks Google prebuilt for. Hyperparameter tuning jobs search for better settings. Model monitoring watches for data drift after deployment. Private networking and encryption keys matter in regulated industries. Feature stores share engineered features between teams.

What comes next. The Mid-level track picks up exactly here, with pipelines, experiment tracking, custom serving containers, autoscaling behaviour, service accounts per job, and CI/CD that promotes a model by moving an alias. The Senior track covers private endpoints, perimeter controls, multi-project design, upgrade strategy, and when to choose a different platform entirely. Alongside it, read the neighbouring guides in this catalogue: Kubeflow for the pipeline model, MLflow for experiment tracking, BentoML and KServe for serving outside a managed platform, and SageMaker to see how Amazon solved the same problems. Seeing two clouds side by side is the fastest way to separate platform ideas from vendor vocabulary.

Keep the habit this guide started: when you build something on a managed platform, write down what it costs to leave running, and then switch it off.

Sources

• Vertex AI documentation (Gemini Enterprise Agent Platform)
• Vertex AI release notes
• Vertex AI deprecations
• Get predictions from a model: overview
• Vertex AI Pipelines introduction
• Vertex AI access control with IAM
• Use the Vertex AI SDK for Python
• Python SDK 2.0 migration guide
• python-aiplatform on GitHub
• google-cloud-aiplatform on PyPI

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Vertex AI. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

Vertex AI is Google Cloud's managed machine-learning platform. You give it a training script and it runs that script on machines it provisions and then releases; you register the resulting model, deploy it to an endpoint that serves predictions over HTTPS and autoscales between a minimum and maximum number of replicas, or run it as an offline batch job. It also hosts generative models such as Gemini behind the same project, billing and permissions.

Then the line that shows you have used it: "everything is regional, a deployed endpoint bills for its machines even when idle, and a job runs as a service account, so most failures are permissions, regions or quotas rather than code."

One more fact signals that your knowledge is current. Google has been folding the Vertex AI name into the Gemini Enterprise Agent Platform. If you say "Vertex AI, which Google now brands as part of the Agent Platform, and the classic ML SDK is still google-cloud-aiplatform", you show you know both names refer to one platform.

The model

Three more nouns round out the vocabulary an interviewer expects. Pipelines are serverless runs of workflows defined with the Kubeflow Pipelines SDK. Experiments record parameters and metrics across runs. Model Garden is a catalogue of Google, open and partner models.

The commands to have ready

Be able to type these from memory in a screen-share. Interviewers watch for whether you reach for the right tool, not whether you remember every flag.

gcloud auth login
gcloud auth application-default login
gcloud config set project PROJECT_ID
gcloud services enable aiplatform.googleapis.com

gcloud ai models list --region=us-central1
gcloud ai endpoints list --region=us-central1
gcloud ai custom-jobs list --region=us-central1
gcloud ai endpoints create --region=us-central1 --display-name=my-ep
gcloud ai endpoints deploy-model ENDPOINT_ID --region=us-central1 \
  --model=MODEL_ID --display-name=DEPLOYMENT_NAME \
  --machine-type=n1-standard-4 --min-replica-count=1 --max-replica-count=3

And the SDK, in the shape you should be able to sketch on a whiteboard:

from google.cloud import aiplatform

aiplatform.init(project="PROJECT_ID", location="us-central1",
                staging_bucket="gs://my-staging-bucket")

job = aiplatform.CustomTrainingJob(display_name="train", script_path="train.py",
                                   container_uri=TRAIN_IMAGE,
                                   model_serving_container_image_uri=SERVE_IMAGE)
model = job.run(replica_count=1, machine_type="n1-standard-4")

endpoint = model.deploy(machine_type="n1-standard-2",
                        min_replica_count=1, max_replica_count=3)
print(endpoint.predict(instances=[[1.0, 2.0, 3.0]]).predictions)

endpoint.undeploy_all()
endpoint.delete()

The last two lines matter as much as the rest. Candidates who show the teardown without being asked signal that they have paid a surprising bill before, or have the sense to avoid one.

For generative calls, the current form uses the Google Gen AI SDK rather than the old Vertex AI generative classes:

from google import genai

client = genai.Client(vertexai=True, project="PROJECT_ID", location="global")
r = client.models.generate_content(model=MODEL_ID, contents="Hello")
print(r.text)

The classic traps, and why they happen

The idle endpoint bill. A deployed endpoint reserves machines for as long as the deployment exists. The platform does not know a lab is over. Billing is per node-hour, so a forgotten endpoint with a single small replica still accrues charges every hour. The remedy is behavioural, undeploy and delete when done, plus a budget alert so a forgotten resource is noticed in days rather than at month end.

Two logins. gcloud auth login authorizes the command-line tool; gcloud auth application-default login creates the credentials that Python libraries read. Someone who has done only the first sees the command line work and the SDK fail with DefaultCredentialsError. They are separate because the tool and your programs are separate clients.

The wrong identity. A job runs as a service account, not as you. You can read your bucket, but the job's account may not be able to. The 403 names a permission, and the fix is to grant a role to the identity that actually made the call, not to your user.

Region drift. Resources are regional, and a model in one region cannot be deployed to an endpoint in another. A bucket in a different location from the job adds latency and sometimes errors. Region is part of the resource name, so a NotFound for something you created usually means you are looking in the wrong place.

Quota surprises. New projects often have little or no GPU quota, so a job that asks for a GPU fails with 429 RESOURCE_EXHAUSTED before it starts. Quota increases take time, so request them early.

Retired model IDs and removed modules. Gemini model versions are retired on a published schedule, and the old vertexai.generative_models family was deprecated in June 2025 with removal scheduled for June 2026. Old tutorials break for that reason. The fix is google-genai and a current model ID from the documentation.

Version mismatch at serving time. A scikit-learn model saved with one library version may fail to load in a serving image with another, and the endpoint reports that the model server never became ready. Match the training and serving library versions and read the container logs.

Common interview questions

What is Vertex AI, and how is it different from running my own server? It is a managed platform for the ML lifecycle. Running your own server means you provision the machine, install dependencies, write the serving code, add authentication and scaling, and patch it. With Vertex AI you supply the model and the code, and the platform handles provisioning, autoscaling between replica limits, logging, IAM integration and billing per use. You trade some control and some cost per unit for far less operational work, and you accept the platform's constraints, such as the request format of the serving containers.

Walk me through taking a model from a script to a prediction. Put data in Cloud Storage or BigQuery and make a staging bucket. Write a training script and run it locally first. Submit it as a CustomTrainingJob with a prebuilt training container, a machine type and a serving container. The job writes the model artifact and registers a Model. Deploy the Model to an endpoint with minimum and maximum replicas, call predict, and when finished undeploy and delete the endpoint. For offline scoring, use batch_predict instead of an endpoint.

What is the difference between a model and an endpoint? A model is a registered artifact plus the serving container that knows how to load it. An endpoint is a serving address. Deploying puts a model onto machine replicas behind an endpoint. One endpoint can have several deployed models with traffic split between them, and one model can be deployed to several endpoints. Only the deployment costs money while idle.

When would you use batch prediction instead of an endpoint? When no one is waiting for the answer. Nightly scoring, one-off labelling and reports all fit. Batch jobs run on temporary machines and you pay only while they run, while an endpoint pays for replicas continuously. Choose an endpoint when latency matters and requests arrive one at a time.

What is the Model Registry for? It is the system of record for models: versions, aliases such as default, labels and lineage back to the training job. It lets a team promote a model by moving an alias and answer "which job produced what is in production" without digging through buckets.

What are Application Default Credentials? A standard way for Google client libraries to find credentials: first an environment variable pointing to a key file, then credentials created by gcloud auth application-default login, then the attached service account when running on Google Cloud. In production on Google Cloud you rely on the attached service account and need no files at all.

A training job fails immediately with a 403. What do you do? Read the message, which names the missing permission or says the API is disabled. If the API is disabled, enable aiplatform.googleapis.com. If it is a permission, determine the identity that ran the job, normally a service account, and grant a minimal role containing that permission, such as access to the bucket or to Artifact Registry. Granting broad Owner access is a weak answer; a narrow role is the strong one.

Why are your resources all in one region? Because most resources are regional, cross-region use adds latency and egress cost, some combinations fail, and data residency follows the region. For employers in the Gulf or Egypt, region choice can be a compliance requirement, so I check which regions support the features I need before designing, not afterward.

How do you control cost? Use a disposable project for experiments and set budget alerts on day one. Undeploy endpoints when not in use. Prefer batch prediction for offline work. Start with small machines and ask for a GPU only when the model needs one. Label resources with owner and purpose so bills can be filtered. For generative calls, remember that cost is per token and that quotas throttle with 429.

How do you call Gemini from Vertex AI today? Install google-genai, create genai.Client(vertexai=True, project=..., location=...), and call client.models.generate_content with a current model ID. I would not use the old vertexai.generative_models classes because they were deprecated and scheduled for removal. I keep the model ID in one constant, because Google retires versions regularly.

What changed in the Python SDK recently? Version 2.0 of google-cloud-aiplatform was a breaking release. Generative modules moved to the Gen AI SDK, and agent tooling moved into a separate package, google-cloud-agentplatform. Classic ML, with aiplatform.init, CustomTrainingJob and Model.deploy, stays in the original package. The package also releases frequently, so I pin versions.

What is a service account and why does a job need one? It is a non-human identity. A job needs one because code running in the cloud must authenticate to read buckets and call other services, and you want it to have only the access that job needs. Using a dedicated service account per workload is the least-privilege habit.

What do AutoML and custom training offer, and which would you choose? AutoML needs no code and suits teams without ML specialists on tabular or image problems where a reasonable model is enough. Some AutoML types have been shut down, such as text and video, so I check the deprecations page. Custom training gives full control of the code and framework, which is what most engineering teams need, so I default to it.

What is Model Garden? A catalogue of models, from Google, open-source and partners, that you can try or deploy from the console. It saves you from packaging a popular model yourself, but you still choose a region, a machine and an endpoint and still pay for the deployment.

Related tools interviewers bring up

You will often be asked to compare. Have one sentence for each.

• SageMaker is Amazon's equivalent platform, with similar concepts under different names; see the SageMaker guide. A good answer says the lifecycle ideas are the same and the differences are in IAM, networking and pricing.
• MLflow tracks experiments and models and is cloud-neutral; see the MLflow guide. It can run beside Vertex AI or replace parts of it.
• Kubeflow Pipelines is the open-source pipeline format Vertex AI Pipelines runs; see the Kubeflow guide.
• KServe and BentoML are serving options you run yourself; see the KServe guide and BentoML guide. They give portability and control; Vertex AI gives less to operate.
• The Gemini API can reach the same models without a Google Cloud project; see the Gemini API guide. The difference is billing, identity and enterprise controls.

If asked "why not just use Kubernetes?", say that you can, and many teams do for control or portability, but that you then own scaling, GPUs, upgrades, security patches and the serving code. A managed platform removes that work, at the price of platform limits and cost per unit.

What interviewers listen for

Interviewers at this level are rarely testing recall of flags. They want evidence that you have done the thing and that you are safe to leave alone with a cloud bill. Three signals matter most.

The first is cost awareness. Mentioning teardown, budget alerts and batch-over-endpoint without being asked is worth more than reciting machine types.

The second is diagnostic order. Strong candidates describe a path: read the error sentence, identify the identity and region, check the logs, and only then change something. Weak candidates describe retrying.

The third is currency. The platform has been renamed and its SDK reorganized within the last year. A candidate who knows that generative code now lives in google-genai, that model IDs are retired on a schedule, and that some products have deprecation dates comes across as someone who reads release notes.

If you have not used the platform at work, say so, and describe your lab project honestly: what you trained, what you deployed, what it cost, and what went wrong. A true story about forgetting an endpoint and putting a budget alert in place afterward is a better answer than a polished one you cannot defend.

Sixty-second self-test

Answer out loud, without looking back.

1. What are the four stages from data to predictions, and which one costs money while idle?
2. Name the two gcloud login commands and what each is for.
3. What two things does a model record point to?
4. What does staging_bucket do, and what happens if you omit it?
5. When do you choose batch prediction over an endpoint?
6. Which error code means a quota was hit, and what are three ways to respond?
7. A job gets 403. Whose identity do you check first?
8. Why is a Gemini model ID kept in a constant?
9. Which package holds CustomTrainingJob, and which holds genai.Client?
10. What two commands remove an endpoint, and in what order?

Answers

1. Data, training, model registry, endpoint. The endpoint, because deployed replicas bill continuously.
2. gcloud auth login for the command-line tool; gcloud auth application-default login for the SDK.
3. The artifact in Cloud Storage and the serving container image.
4. It is the bucket where the SDK uploads your script and where outputs are staged; without it, custom training will not start.
5. When no one is waiting for the answer and volume or schedule fits an offline job.
6. 429; request a quota increase, choose another region, or use a smaller machine or add retries with backoff.
7. The service account the job runs as, not your own user.
8. Because models are retired on a schedule and the ID must be changed in one place.
9. google-cloud-aiplatform and google-genai.
10. Undeploy all models first, then delete the endpoint.

If you missed more than two, reread the detailed track's training, endpoint and errors sections, then run the end-to-end project once more.

What the next two tracks add

Mid-level takes every one of those topics further: pipelines built with the Kubeflow Pipelines SDK and scheduled or triggered, experiments and metadata, custom serving containers and the environment variables the platform passes them, hyperparameter tuning, autoscaling and traffic splitting between model versions, service accounts per job, and promotion of models through aliases in CI.

Senior then covers what you own when Vertex AI is a platform for a team: private networking with Private Service Connect, perimeter controls and customer-managed encryption keys, project structure and multi-tenancy, quota and capacity planning, upgrade strategy given weekly SDK releases and dated product deprecations, governance and audit logging, incident playbooks, and honest answers to when to choose a different platform.

Part one of three. Most beginner trouble on Vertex AI comes from three habits: forgetting that a deployed endpoint keeps billing, forgetting that jobs run as a service account rather than as you, and forgetting that everything lives in a region. Start with the table, then read the longer notes on the mistakes that cost the most, then the habits and the checklist.

Common errors at this level

Mistakes beginners actually make

Each of these has a symptom, a cause and a fix. They are the ones that cost real time and money.

1. Leaving an endpoint deployed overnight

Symptom. A few days after the course lab, a billing email arrives, or the cost report shows a flat daily charge for Vertex AI even though you did nothing.

Cause. A deployed model runs on machine replicas that are billed per node-hour. The platform has no idea that the lab is over, and an endpoint with zero traffic costs the same as a busy one. Many people delete the model or close the notebook and assume the serving stopped, but the endpoint is a separate resource.

Fix. Run endpoint.undeploy_all() and then endpoint.delete() the moment you are done. Confirm with gcloud ai endpoints list --region=YOUR_REGION. Make it a habit to run that listing at the end of every session, for every region you have used. And set a budget alert on day one so that if you do forget, you find out in days rather than weeks.

2. Mixing up the two logins

Symptom. gcloud ai models list works in the terminal, but your Python script fails with DefaultCredentialsError.

Cause. There are two separate credential stores. gcloud auth login authorizes the command-line tool. Application Default Credentials, which client libraries search for automatically, come from gcloud auth application-default login. Setting up one does not set up the other.

Fix. Run both commands once on a new machine. If your script then acts as an account you did not expect, run gcloud auth list to see the command-line account and think about which account the ADC belongs to, since they can differ.

3. Downloading a service-account key "to make it work"

Symptom. A tutorial says to create a JSON key and set GOOGLE_APPLICATION_CREDENTIALS. It works. Months later, a security alert arrives, or someone finds the file in a repository.

Cause. A key file is a long-lived password. It does not expire unless you delete it, it can be copied silently, and public repositories are scanned continuously by tools that look for exactly this.

Fix. On your laptop, use gcloud auth application-default login. On Google Cloud, attach a service account to the workload so no file is needed. If your organization has CI, ask for keyless authentication with workload identity federation rather than a stored key. Add *.json key patterns and any credentials folders to .gitignore, and never commit them.

4. Giving yourself Owner and moving on

Symptom. Everything works in your personal project. At work, with a narrower role, the same code produces 403 after 403.

Cause. In a personal project you are the Owner and every permission is implicit. In a company project you may hold only something like the Vertex AI User role (roles/aiplatform.user), and the training job's service account needs its own permissions on buckets and repositories.

Fix. Learn which permissions your workflow needs by reading the error, which names the one that is missing. Practise with a narrow role in a lab project so that the first time you meet a permission error is not the first day of a new job. Then grant the smallest role that contains the named permission, to the identity that actually ran the call.

5. Letting the region drift

Symptom. A model you uploaded cannot be found, a deployment complains about a different location, or a job runs slowly and costs more because its data sits far away.

Cause. Resources are regional, and the default in a script or command may not be the region you used before. aiplatform.init(location=...) sets the region for later calls; --region does it for gcloud. One misplaced default produces a resource in one place and a lookup in another.

Fix. Put the region in one variable at the top of every script and in an exported shell variable for commands. Create your bucket in the same region. When something is "not found", check the region before anything else. If data residency matters to your employer, which for many Gulf and Egyptian organizations it does, check the locations documentation for the region you need before you build.

6. Skipping the local run

Symptom. A cloud job takes ten minutes to start and then fails on a typo in line 3 of the script. You fix it, resubmit, and wait again.

Cause. Starting a managed job involves provisioning a machine and pulling a container image. That overhead is paid on every submission, however small the mistake.

Fix. Run python train.py locally on a tiny sample before you submit anything. Add a command-line flag or an environment variable that switches the script to a small subset of data. A bug caught on your laptop costs seconds.

7. Copying old tutorial code

Symptom. ModuleNotFoundError: No module named 'vertexai.generative_models', or a model name that returns 404.

Cause. The platform has changed quickly. The generative modules that lived inside the old SDK were deprecated in June 2025 and scheduled for removal in June 2026. Gemini model versions are retired on a published schedule. The product itself has been renamed. A tutorial from eighteen months ago can be wrong in several places at once.

Fix. Check the date of a tutorial before you follow it. Prefer official documentation and the release notes. Use google-genai for generative calls, keep the model ID in one constant, and pin the SDK version in requirements.txt so an upgrade is a decision you make rather than something that happens to you. Python 3.9 and older are deprecated, so use Python 3.10 or newer.

8. Using a stale container image

Symptom. A job fails at start-up with an image-pull error, or a deployment fails because the serving image cannot load the model.

Cause. The container_uri or serving image URI was copied from an old source, so the tag has been retired, or the framework version in the image differs from the one you trained with.

Fix. Copy image URIs from the current prebuilt-container pages in the documentation, and use matching versions of the framework for training and serving. Record the exact versions in your README. If a model loads on your laptop but not on the endpoint, compare library versions first.

9. Not reading the logs

Symptom. A job fails and the only thing you have is a red status in the console and a generic message.

Cause. The status is a summary. The detail is in Cloud Logging, where the platform and your script both write.

Fix. Open the job in the console and follow its logs link. Read the first error, not the last line. For endpoints, look at the serving container's output. Make your own script print what matters: the shapes of the data, the metrics, and the path where the model is saved. Printing is your only window into a machine you cannot log into.

10. Sending the wrong shape to an endpoint

Symptom. 400 from predict, with a message from the model server about dimensions or a missing field.

Cause. Each serving container expects a specific request format. The scikit-learn image expects a list of feature rows, for example, and other images have their own conventions.

Fix. Read the format described for the prebuilt container you chose, and test with one instance first. Remember that the error is from the model server inside the endpoint, so it often says exactly what it expected.

Habits worth adopting on day one

Use one project per course, and delete it at the end. A disposable project turns "did I clean up everything?" from a hard question into a single click. Do the same for each project you try. For work, use the environment-per-project pattern your team has chosen.

Set a budget alert before you create anything. Pick thresholds at fifty, ninety and one hundred percent of what you are willing to spend. It is an alert and not a cap, so you still must act on it, but it turns a month-end shock into a day-two email.

Put project, region, bucket and model ID at the top of every script. A script with these four names in one place is easy to change, easy to share and easy to review. A script with them scattered through the code is where region drift is born.

Name and label everything. train-sklearn-toy beats test2. A label such as owner=your-name and purpose=lab lets you filter the console and the billing report. At work, labels are how finance asks which team spent what.

Pin your versions. Put google-cloud-aiplatform==X.Y.Z in requirements.txt with the exact version you tested. The package releases frequently, and the 2.0 release changed where several features live. Upgrade deliberately, in a branch, and read the release notes first.

Test small, locally, first. Every minute spent making the script correct on a laptop is a minute you do not pay start-up overhead for.

Read the whole error sentence. Google errors usually name the permission, the resource, the API or the quota metric. The answer is in the message more often than beginners expect.

Write down what you did. A README with the region, versions, commands and costs makes your lab reproducible, and it is the best interview artifact a beginner can have. Reviewers love "here is what went wrong and how I fixed it".

Stop notebooks. A Workbench instance bills while it runs. Stop it when you leave for the day. The stop button is on the instance list.

Use the neighbours. Experiment tracking, pipelines and serving each have good open-source alternatives, and comparing them teaches you which ideas are Google's and which are universal. See the MLflow guide, the Kubeflow guide and the SageMaker guide.

Practice cards

Work through these once. Each is a small exercise you can finish in under thirty minutes.

A recipe for a calm lab session

A short routine removes most of the anxiety around cloud labs, and it is worth turning into muscle memory.

Start. Open a terminal, activate the virtual environment, run gcloud config list to confirm the project, and gcloud auth list to confirm the account. Say the region out loud. It sounds silly and it prevents errors.

During. Keep a scratch file open where you paste every resource name the platform prints: job names, model resource names, endpoint IDs. Copy them as they scroll past, because the console is slow and a scrollback is not a record.

Before you stop for a break. Ask three questions. Is there an endpoint deployed? Is a notebook instance running? Is a job still running? If the answer to any is yes and you will not use it within the hour, stop or delete it.

At the end. Run the three listing commands one more time, in each region you used:

gcloud ai endpoints list --region=us-central1
gcloud ai custom-jobs list --region=us-central1
gcloud ai models list --region=us-central1

Endpoints should be empty. Custom jobs should be in a finished state, not running. Models are cheap to keep, so a list there is fine, but delete test models you do not need so the registry stays readable.

Last, glance at the billing report and write one line in your README about the day's cost. Two weeks of those lines teach you what the platform actually costs, which is knowledge many working engineers do not have.

Reading a failed job, step by step

Because this is where beginners lose the most time, here is a repeatable method for a failed custom training job.

1. Look at the state and the error summary in the console. Is it FAILED, or is it stuck pending? Stuck pending often points to quota or capacity, while failed usually means your code or its environment.
2. Open the logs. Scroll to the first error, not the last. A traceback near the end is often a consequence of something earlier.
3. Classify. If the first error is about an image, it is a pull problem. If it is ModuleNotFoundError, a package is missing from requirements. If it is a permission message, it is IAM and the identity is the job's service account. If it is your own traceback, reproduce it locally.
4. Change one thing. Resubmit after a single change so that you know what fixed it.
5. Write the cause in your notes. The next time the error appears, you will recognise it in seconds.

The same method works for a failed deployment, with the serving container's logs in place of the training logs, and for a failed batch job, with the batch job's logs.

Checklist

Before you consider a Vertex AI lab done, tick every box.

• I created a project just for this exercise and linked billing.
• I set a budget alert with more than one threshold.
• I enabled aiplatform.googleapis.com and can say how.
• I ran both gcloud auth login and gcloud auth application-default login.
• My project, region, bucket and model ID are in one place at the top of my scripts.
• My bucket is in the same region as my jobs.
• I ran the training script locally before submitting it.
• My container image URIs came from the current documentation.
• I pinned the SDK version in requirements.txt and use Python 3.10 or newer.
• I read the job's logs and found my own printed output.
• I deployed an endpoint, called it, and tore it down with undeploy_all() and delete().
• I confirmed with gcloud ai endpoints list in every region I used.
• I stopped or deleted every notebook instance.
• I used google-genai, not the removed generative modules, and kept the model ID in one constant.
• I did not create, download or commit a service-account key file.
• I wrote a README with the region, versions, commands, cost and what surprised me.

If every box is ticked, you are ready for the Mid-level track, which adds pipelines, experiment tracking, custom containers and promotion of models by alias. If a box is not ticked, the habit behind it is the one to fix first.

This is part two of three. It picks up where Beginner ended and takes the platform further: you already submit training jobs, upload a model and call an endpoint, so this level is about doing those things in a way that survives a team, a CI system and a quarterly SDK upgrade.

A naming note first, because you will meet it in every tutorial you open. Google has been folding the product name "Vertex AI" into Gemini Enterprise Agent Platform, and the console and docs now use the new name. The concepts, the aiplatform.googleapis.com service and the resource names this guide uses are the ones you already know, so this guide keeps saying Vertex AI for the classic machine-learning platform and mentions the new name only where it changes code.

Where this picks up

Each section opens with the problem it solves and ends with a Try it that you can run against a real project. Run them in a sandbox project with a budget alert, because several of them create billable resources, and the guide tells you when to delete them.

The resource model, precisely

Beginner treated the project and region as two arguments you pass to init. At work they are the most consequential choices you make, because they decide who can see your data, where it physically sits and which features exist.

Almost every Vertex AI resource is regional. A model in europe-west4 is invisible from us-central1: gcloud ai models list --region=us-central1 shows only that region's models, and an endpoint can only serve a model that lives in its own region. Each resource has a full name that encodes this:

projects/123456789/locations/us-central1/models/987654321
projects/123456789/locations/us-central1/endpoints/555
projects/123456789/locations/us-central1/models/987654321@3

The last form is a model with a version suffix. Names use the project number, not the project ID, and the SDK accepts either a full name or the bare numeric ID once init has supplied the project and location. When you copy a resource name between a log line, a script and a Terraform file, the project number versus project ID difference is the usual reason a lookup returns 404 NotFound.

The region also governs your data. The staging bucket, the training data, the container image and the Vertex resources should all be in the same region, or in a multi-region that includes it. A bucket in a different location from the job does not always fail outright, but it adds latency and egress cost, and for a Gulf or Egyptian employer it may break a residency commitment. Which regions exist for which features is documented on the locations page, and it moves, so confirm that a feature is offered in the region your compliance team approved before you design around it. Many generative features default to the global location or to US and EU regions, which is worth knowing before a regulated workload commits to a design.

The SDK resolves its settings in a predictable order. Explicit arguments to a constructor win, then aiplatform.init() values, then environment defaults. Keep that in mind when a script behaves differently in CI: a missing init call quietly falls back to defaults and you get a "wrong project" error far from the cause.

from google.cloud import aiplatform

aiplatform.init(
    project="acme-ml-dev",
    location="europe-west4",
    staging_bucket="gs://acme-ml-dev-europe-west4-staging",
    experiment="churn-v2",
)

Name the staging bucket with the region in it. It looks fussy, but it makes a mismatch visible in every log line and every code review.

Identity: who a job really runs as

The most common mid-level incident on this platform is a permission error that appears only after code moves from a laptop to a pipeline. It happens because the identity changes and nobody noticed.

Three identities matter. Your user credentials, created by gcloud auth application-default login, are what the SDK uses on your laptop. A custom service account is the identity your training code or pipeline step runs as, and you choose it. The Vertex AI service agent is a Google-managed account that the platform itself uses to do work for you, such as pulling your training image from Artifact Registry. Confusing them produces the two classic errors: code that works for you and fails in a job, and a job that cannot pull its image.

The rule is simple. Every job, pipeline and endpoint that matters runs as a dedicated custom service account with the smallest set of roles that works, never as a person and never as the default compute account, which tends to carry broad project permissions. Submitting under that account requires two separate grants, and missing either one is a frequent time sink:

1. The service account needs the roles its code uses, for example read access on the data bucket and roles/aiplatform.user to call Vertex AI.
2. You, or your CI identity, need roles/iam.serviceAccountUser on that service account, so you are allowed to act as it. Without it, submission fails with an error mentioning iam.serviceAccounts.actAs.

gcloud iam service-accounts create trainer-churn \
  --display-name="Churn training and pipelines"

gcloud projects add-iam-policy-binding acme-ml-dev \
  --member="serviceAccount:trainer-churn@acme-ml-dev.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

gcloud iam service-accounts add-iam-policy-binding \
  trainer-churn@acme-ml-dev.iam.gserviceaccount.com \
  --member="user:aya@example.com" \
  --role="roles/iam.serviceAccountUser"

Grant bucket access on the specific bucket rather than the project, so a leaked token reaches one dataset and not all of them. For reading secrets, give the account roles/secretmanager.secretAccessor on the individual secret.

The image-pull case is the service agent's. If a custom job fails with an image pull permission error, the fix is on the registry side: the Vertex AI service agent in your project needs read access to the Artifact Registry repository that holds the image. Cross-project registries make this common, because the repository lives in a platform team's project and the service agent lives in yours.

To prove which identity code is using, print it instead of guessing:

import google.auth

creds, project = google.auth.default()
print(type(creds).__name__, getattr(creds, "service_account_email", "user credentials"), project)

Inside a job this prints the custom service account's email. On your laptop it prints user credentials. If both runs behave differently, you have your answer.

Custom training done properly

The Beginner path used a script and a prebuilt container. That is the right start, and it stops being right when you need a library the prebuilt image lacks, a system package, or a build you can reproduce a year later. At that point you move to a custom container, and you pick the lowest-level API that fits.

There are three levels, and knowing them prevents rewrites:

CustomJob is the underlying primitive. It takes a list of worker pool specs: a machine type, a replica count and a container for each pool. The first pool is the chief, and additional pools are workers, parameter servers or evaluators. Everything else is a convenience layer on top.

from google.cloud import aiplatform

aiplatform.init(project="acme-ml-dev", location="europe-west4",
                staging_bucket="gs://acme-ml-dev-europe-west4-staging")

job = aiplatform.CustomContainerTrainingJob(
    display_name="churn-train",
    container_uri="europe-west4-docker.pkg.dev/acme-ml-dev/ml/churn-train:2026-10-01-a1b2c3d",
    model_serving_container_image_uri="europe-west4-docker.pkg.dev/acme-ml-dev/ml/churn-serve:2026-10-01-a1b2c3d",
)

model = job.run(
    args=["--train-uri=gs://acme-ml-dev-data/churn/2026-09/", "--epochs=20"],
    replica_count=1,
    machine_type="n1-standard-8",
    service_account="trainer-churn@acme-ml-dev.iam.gserviceaccount.com",
    environment_variables={"LOG_LEVEL": "INFO"},
    labels={"team": "growth", "model": "churn", "env": "dev"},
)

Several details here are deliberate. The image tag combines a date and a commit hash, so a model version can always be traced to the code that produced it. The image is in Artifact Registry in the same region; the older gcr.io Container Registry is no longer the target for new work. The service account is explicit. And the labels are set on the job, because labels are the only practical way to attribute cost later, as the cost section explains.

How the training code finds its inputs and outputs matters as much as the call. Vertex AI injects environment variables into the container. The ones to use are the output locations for the model and checkpoints, which the platform sets so your code writes to Cloud Storage without hard-coding a bucket. Check the training-code requirements page for the exact variable names in your SDK version rather than copying them from a blog post, because they are what ties your script to the platform. Writing the model artifact to the location the platform provides is what lets job.run() return a registered Model instead of leaving you to hunt through buckets.

Checkpointing is the habit that separates a mid-level engineer from a beginner here. A worker can be preempted, a spot VM can be reclaimed, and a container can be rescheduled. If your training script writes a checkpoint to Cloud Storage every N steps and resumes from the latest one on startup, interruptions cost minutes rather than hours. Cheaper capacity such as spot VMs is only worth using when your job can restart from a checkpoint.

For tuning, HyperparameterTuningJob wraps a CustomJob as a trial template and runs many trials, each given parameter values on its command line. Your script must accept those parameters as arguments and report the metric back in the way the tuning docs require; a script that trains fine but never reports the metric yields trials that all finish with no result.

Building containers and the serving contract

Training containers are mostly your own business. Serving containers are a contract, and breaking the contract is why a deploy sits for fifteen minutes and then fails with a message that the model server never became ready.

When Vertex AI starts a serving container it expects an HTTP server that satisfies a small interface. The platform tells your server which port and routes to use through environment variables, and your server must honour them: AIP_HTTP_PORT for the port, AIP_HEALTH_ROUTE for the health check path, AIP_PREDICT_ROUTE for the prediction path and AIP_STORAGE_URI for where the model artifacts live. The server must answer the health route with a success status once it is ready, and answer the predict route with a JSON body shaped as {"instances": [...]} returning {"predictions": [...]}.

import os
from fastapi import FastAPI, Request
from model_loader import load_model

app = FastAPI()
model = load_model(os.environ.get("AIP_STORAGE_URI", "/model"))

@app.get(os.environ.get("AIP_HEALTH_ROUTE", "/health"))
def health():
    return {"status": "ok"}

@app.post(os.environ.get("AIP_PREDICT_ROUTE", "/predict"))
async def predict(request: Request):
    body = await request.json()
    return {"predictions": model.predict(body["instances"]).tolist()}

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["sh", "-c", "uvicorn serve:app --host 0.0.0.0 --port ${AIP_HTTP_PORT:-8080}"]

A few points make the difference between a container that deploys and one that does not. Bind to 0.0.0.0, not localhost, or the platform's health checker cannot reach you. Load the model before the health route reports ready, or the first real request lands on a server that is still loading. Do heavy imports and model loading at startup, never per request. And remember that the container runs as a normal process whose logs go to stdout and stderr, which Vertex AI forwards to Cloud Logging, so a crash on startup is visible there and nowhere else.

Local testing beats debugging a failed deploy. Run the exact image on your machine, set the same environment variables, and send it the request shape you will use in production:

docker run --rm -p 8080:8080 -e AIP_HTTP_PORT=8080 \
  -e AIP_HEALTH_ROUTE=/health -e AIP_PREDICT_ROUTE=/predict \
  -v "$PWD/model:/model" -e AIP_STORAGE_URI=/model churn-serve:test

curl -s localhost:8080/health
curl -s -X POST localhost:8080/predict -H 'content-type: application/json' \
  -d '{"instances": [[1.0, 2.0, 3.0]]}'

If either curl fails, the endpoint will fail too, and now you are debugging on your own machine. For image hygiene, build the image the way the Docker guide describes: slim base, dependencies before source, a non-root user and a pinned tag. If your team prefers a serving framework, BentoML and FastAPI both package cleanly into a container that meets this contract.

Pipelines: KFP on a managed runner

Notebooks and one-off jobs do not give you repeatability, an audit trail or a way to rerun only the part that failed. Vertex AI Pipelines does. It runs pipelines written with the Kubeflow Pipelines SDK (KFP) as a serverless service, with no cluster for you to operate. If you know Kubeflow or Airflow, the mental model transfers: a pipeline is a DAG, a component is a step that runs in its own container, and a run is one execution.

The workflow has three stages: define, compile, submit. Compilation turns your Python into a YAML file that describes the graph, which makes it an artifact you can version, review and submit from CI without importing your authoring code.

from kfp import dsl, compiler
from google.cloud import aiplatform

@dsl.component(base_image="python:3.11", packages_to_install=["pandas", "scikit-learn"])
def train(train_uri: str, model: dsl.Output[dsl.Model], metrics: dsl.Output[dsl.Metrics]):
    import pandas as pd
    from sklearn.linear_model import LogisticRegression
    import joblib
    df = pd.read_csv(train_uri)
    clf = LogisticRegression().fit(df.drop(columns="y"), df["y"])
    joblib.dump(clf, model.path)
    metrics.log_metric("train_accuracy", float(clf.score(df.drop(columns="y"), df["y"])))

@dsl.pipeline(name="churn-train", pipeline_root="gs://acme-ml-dev-europe-west4-staging/pipeline-root")
def pipeline(train_uri: str):
    train(train_uri=train_uri)

compiler.Compiler().compile(pipeline, "churn-train.yaml")

aiplatform.PipelineJob(
    display_name="churn-train",
    template_path="churn-train.yaml",
    parameter_values={"train_uri": "gs://acme-ml-dev-data/churn/2026-09/train.csv"},
    enable_caching=True,
).submit(service_account="trainer-churn@acme-ml-dev.iam.gserviceaccount.com")

Three things to understand about how it works.

Artifacts versus parameters. Small values such as strings, numbers and booleans are parameters passed by value. Files such as datasets and models are artifacts, and what moves between steps is a path under pipeline_root in Cloud Storage, not the data itself. Output[dsl.Model] gives your code a local file path that the runtime uploads for you. This is why pipeline_root is mandatory and why it should be a bucket your service account can write.

Each step is its own container. A lightweight Python component installs its packages_to_install at start, so two steps can use different library versions, and a slow pip install is paid per step per run. For heavy steps, build the dependencies into an image and pass it as base_image, which is faster and, importantly, reproducible. If a step needs a particular machine, set its resources on the task rather than the pipeline; a CPU-bound preprocessing step and a GPU training step should not share a machine type.

Caching. With enable_caching=True, a step whose inputs and component definition match a previous successful run is not executed again; its previous outputs are reused. This makes iteration fast and has one sharp edge. The cache key covers the step's inputs and definition, not the contents of the data behind a path. If you pass the same gs://.../latest/ string and the files under it change, the step is served from cache and you train on yesterday's data. Pass versioned paths, such as a date-stamped folder, or turn caching off for the steps that read mutable inputs.

The Google Cloud Pipeline Components library provides ready-made steps that wrap Vertex operations: upload a model, create an endpoint, deploy, run a batch prediction. Using them keeps the pipeline declarative and makes lineage show up in the console, at the price of one more dependency to pin. Writing your own component that calls the SDK is equally valid and often clearer for one-off operations.

Operational features you should know exist: a failure policy that decides whether one failing branch stops the whole run or lets independent branches finish, Secret Manager references so credentials never appear in the YAML, per-step machine types, and scheduling, so the pipeline runs weekly without a human, with Pub/Sub able to trigger runs from events such as new data landing.

The Model Registry: versions, aliases and promotion

Without a registry, "the model in production" is a path someone remembers. The Model Registry gives every trained model a name, an ordered list of versions, free-form labels and aliases, which are movable pointers to a version, so promotion becomes a metadata operation instead of a copy.

A model has a resource name, and each version is addressed as MODEL_ID@VERSION. Uploading a new version of an existing model uses parent_model, and you can attach aliases at upload time:

model_v2 = aiplatform.Model.upload(
    display_name="churn",
    parent_model="projects/123456789/locations/europe-west4/models/987654321",
    artifact_uri="gs://acme-ml-dev-europe-west4-staging/models/churn/2026-10-01/",
    serving_container_image_uri="europe-west4-docker.pkg.dev/acme-ml-dev/ml/churn-serve:2026-10-01-a1b2c3d",
    version_aliases=["candidate"],
    labels={"git_sha": "a1b2c3d", "data": "2026-09"},
)
print(model_v2.resource_name, model_v2.version_id)

The design that works in practice has three conventions. First, every version carries labels that point back to its inputs: the commit hash, the dataset version and the pipeline run. Six months on, when someone asks why the model got worse, those labels are the difference between an answer and an archaeology project. Second, aliases describe lifecycle, not content: candidate for a version that passed training, champion for the version serving production. A deployment script asks for the champion alias and never a version number. Third, promotion is a gate: the alias moves only after an evaluation step in the pipeline has compared the candidate against the current champion on the same held-out data and a person or a rule approved it.

The registry integrates with the rest of Vertex AI: a registered model is what you deploy to an endpoint or hand to a batch prediction job, and the lineage view shows which pipeline run and dataset produced it. If your team already uses MLflow, do not run two competing sources of truth. Either use MLflow for experiment tracking and the Vertex Model Registry for deployable models, with the MLflow run ID stored as a label, or standardise on one. The failure mode to avoid is two registries that disagree about which version is live.

Beyond one model, the registry is where governance starts. Restrict who can move the champion alias through IAM on the project, since changing an alias changes what a deployment script serves. In a multi-project setup, a common pattern is training in a dev project, registering there and copying the approved artifact into a prod project's registry, so production never trusts a path in a project that data scientists can write to.

Endpoints: deploying, scaling and rolling out safely

An endpoint is a stable address; a deployed model is a model version running on replicas behind it, with a traffic split across whatever is deployed. Understanding that split is the key to safe releases, because it means you never replace a model in place. You deploy beside it and shift traffic.

First, the endpoint type. The documentation recommends dedicated public endpoints for public access and Private Service Connect for private access, over the shared public endpoint. A dedicated endpoint gives your traffic its own serving path instead of the shared one, and a private endpoint keeps requests off the public internet. Choose when you create it, because the type is not something you casually switch afterwards. Check the current endpoint documentation for the exact creation arguments in your SDK version.

Deployment settings carry real cost and availability consequences:

endpoint = aiplatform.Endpoint.create(display_name="churn-prod")

model.deploy(
    endpoint=endpoint,
    deployed_model_display_name="churn-v3",
    machine_type="n1-standard-4",
    min_replica_count=2,
    max_replica_count=6,
    service_account="serve-churn@acme-ml-dev.iam.gserviceaccount.com",
    traffic_percentage=0,
    labels={"team": "growth", "model": "churn", "env": "prod"},
)

Replicas and autoscaling. min_replica_count is a promise you pay for around the clock: those replicas run, and bill per node-hour, even with zero traffic. max_replica_count is the ceiling the autoscaler may reach as utilisation rises. Set the minimum to at least two for anything that matters, so one replica failing or being updated does not take the endpoint down, and size the maximum against your quota, not your hopes. GPU quota is regional and is the usual reason a scale-up is slow or refused.

Cold starts and 503s. New replicas must pull the image, start the container and load the model before they pass the health check. During a traffic spike, requests can arrive before capacity does, producing 503 Service Unavailable or deadline errors. The mitigations are a realistic minimum, a lightweight model load, and a client with retries and exponential backoff with jitter. Retrying without backoff turns a short capacity gap into a self-inflicted outage.

Payload limits. Online prediction has a request size limit, on the order of megabytes. Large inputs, such as images or long text batches, should move through Cloud Storage and batch prediction, or be sent as references your serving container fetches itself.

A canary rollout follows directly from the traffic split. Deploy the new version with traffic_percentage=0, test it by sending requests that target that deployed model directly, then shift a small share of traffic and watch error rate and latency before moving the rest.

# Deploy the new version next to the current one, taking 10% of traffic
model_v3.deploy(endpoint=endpoint, machine_type="n1-standard-4",
                min_replica_count=2, max_replica_count=6, traffic_percentage=10)

# Roll back instantly by moving 100% back to the previous deployed model
endpoint.update(traffic_split={"1111111111": 100, "2222222222": 0})

The keys of traffic_split are deployed model IDs, shown in the console and in endpoint.list_models(), not model version numbers. Mixing the two up is a common mistake. Rollback is a traffic change measured in seconds, which is the entire argument for keeping the previous deployment running until the new one has proven itself.

Finally, the cost trap: an idle endpoint still costs money. A forgotten test endpoint with two GPU replicas is a standard line item in a surprising bill. Every lab and every demo ends with an undeploy and a delete:

endpoint.undeploy_all()
endpoint.delete()

Batch prediction, and choosing between modes

Not every prediction needs a live endpoint. If you score a million customers every night, an always-on endpoint wastes money between runs and struggles with the burst. Batch prediction runs a job that reads inputs from Cloud Storage or BigQuery, scores them on machines it provisions, writes outputs to a destination and then releases the machines. Nothing idles.

batch = model.batch_predict(
    job_display_name="churn-nightly-2026-10-01",
    gcs_source="gs://acme-ml-dev-data/churn/score/2026-10-01/*.jsonl",
    gcs_destination_prefix="gs://acme-ml-dev-data/churn/scored/2026-10-01/",
    machine_type="n1-standard-8",
    starting_replica_count=2,
    max_replica_count=10,
    labels={"team": "growth", "model": "churn"},
)
batch.wait()
print(batch.state)

The decision between modes is a latency and cost question, and it is worth stating the rule out loud in design reviews.

Two operational points apply. Batch jobs fail in ways endpoints do not: one malformed input line can fail a record or the job, depending on configuration and format, so validate and sample the input before submitting a large job, and check the output folder for error files as well as the job state. And because a batch job's cost is proportional to wall-clock time on the machines you request, machine type and replica counts are a direct cost lever; a bigger machine that finishes in half the time often costs the same and returns sooner.

If your team stores features in BigQuery, batch prediction reading directly from a BigQuery table and writing results back is often simpler than exporting files, and it keeps the output queryable by analysts. The newer Feature Store is BigQuery-backed for the same reason. Avoid building new work on the legacy Feature Store, which is deprecated with a shutdown date in February 2027.

Experiments and tracking what you ran

Comparing runs from memory does not scale past two people. Vertex AI Experiments records parameters, metrics and artifacts per run so you can compare them in the console or in a DataFrame. It sits on top of ML Metadata, the same lineage store that pipelines write to.

aiplatform.init(project="acme-ml-dev", location="europe-west4", experiment="churn-v2")

aiplatform.start_run("lr-0-01-depth-6")
aiplatform.log_params({"lr": 0.01, "max_depth": 6, "data": "2026-09"})
for epoch, (loss, auc) in enumerate(train_loop()):
    aiplatform.log_time_series_metrics({"loss": loss}, step=epoch)
aiplatform.log_metrics({"auc": 0.91})
aiplatform.end_run()

df = aiplatform.get_experiment_df("churn-v2")
print(df.sort_values("metric.auc", ascending=False).head())

Name runs so a human can tell them apart, record the dataset version alongside the hyperparameters, and log the metric you will decide on, not every metric you can compute. Inside a pipeline, metrics emitted through dsl.Metrics outputs appear on the run and can be compared across runs without extra code.

The honest comparison with other trackers is worth knowing. Experiments is convenient because it needs no server and inherits Google Cloud IAM, which matters when data must not leave the project. Teams that already run MLflow or Weights & Biases have richer comparison interfaces and often keep using them. What matters is that one tool is the record of truth and that every registered model version links back to the run that produced it, via the labels from the registry section. For production monitoring, the question changes from "which run was best" to "is the live model still healthy", which is a job for model monitoring and for Evidently style drift reports, not for experiment tracking.

The SDK, generative calls and surviving upgrades

Vertex AI's Python library changes quickly, and a major version in 2026 changed the rules. Treating the SDK as a pinned dependency rather than something you upgrade casually is a mid-level habit that pays for itself.

The state at the time of writing is this. The google-cloud-aiplatform package is on the 2.x series, with releases arriving roughly weekly, and it requires Python 3.10 or newer; Python 3.9 and earlier are deprecated. The 2.0 release was the breaking one:

• The generative modules inside the old library, vertexai.generative_models, vertexai.language_models, vertexai.vision_models, vertexai.tuning and vertexai.caching, were deprecated in June 2025 and scheduled for removal in June 2026. New code calls Gemini through the Google Gen AI SDK, the google-genai package.
• The agent features moved into a separate package, google-cloud-agentplatform, imported as agentplatform. It does not contain the classic machine-learning classes, and it no longer reads project and location from aiplatform.init().
• The classic machine-learning surface that this guide uses, aiplatform.init, CustomJob, Model, Endpoint and pipelines, stays in google-cloud-aiplatform.

A current Gemini call therefore looks like this, with the project, location and model chosen by you:

from google import genai

client = genai.Client(vertexai=True, project="acme-ml-dev", location="global")
response = client.models.generate_content(
    model=GEMINI_MODEL,   # read the current model ID from the model versions page
    contents="Summarise this support ticket in one sentence: ...",
)
print(response.text)

Notice that GEMINI_MODEL is a variable, not a literal. Model identifiers are retired on a schedule: Gemini 1.5 access ended in September 2025 and several partner models have shutdown dates in 2026, so a hard-coded ID in a script is a future outage. Put the model name in configuration, read the current list from the model versions page, and handle a 404 NotFound for a publisher model as "this model was retired", not as a transient error. If you build agents rather than single calls, the Google ADK guide and the Gemini API guide cover the application side.

For upgrades, three practices keep you safe. Pin exactly in a lock file, for example google-cloud-aiplatform==2.3.0, and update on purpose. Turn warnings into errors in CI with python -W error::DeprecationWarning for your test run, so a deprecation shows up in a pull request instead of a production import failure. And read the release notes and deprecations page on a schedule, since the platform retires components with dated notices: for example the legacy Feature Store and Vertex Explainable AI both have shutdown dates in March 2027 or earlier. A recurring calendar item to scan the deprecations page costs thirty minutes a quarter.

google-cloud-aiplatform==2.3.0
google-genai==<pinned version>
kfp==2.<pinned minor>
google-cloud-pipeline-components==<pinned version>

CI/CD and configuration management

A pipeline someone runs from a notebook is a prototype. A pipeline that runs because a pull request merged is a system. The shape that works is the same as for any other deployable: build artifacts in CI, promote them through environments, and make the environment the only thing that differs.

A typical flow has four jobs. On every pull request, run unit tests, lint, and compile the pipeline to catch definition errors cheaply. On merge, build and push the training and serving images tagged with the commit hash. Then submit the compiled pipeline to the dev project, where it trains, evaluates and registers a candidate. After approval, promote the same artifacts to prod: the pipeline there deploys the already-registered model version, instead of retraining.

Authentication from CI should use workload identity federation, not a downloaded service account key. The CI provider presents a short-lived token, and Google Cloud exchanges it for access as a service account. For GitHub Actions the google-github-actions/auth action implements it, so no long-lived key sits in a repository secret waiting to leak. Give CI a deployer identity that may submit pipelines and act as the pipeline's runtime service account, and nothing broader.

name: train
on:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write
jobs:
  submit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: ${{ vars.WIF_PROVIDER }}
          service_account: ${{ vars.CI_SA }}
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements-ci.txt
      - run: python submit_pipeline.py --env dev --image-tag ${{ github.sha }}

Configuration is the other half. Keep environment differences, meaning project, region, bucket, service accounts and machine sizes, in one config file per environment that the submission script reads, instead of scattering if env == "prod" through the code. Treat the config the way the Docker guide treats environment variables: the artifact is identical everywhere and only the configuration varies.

# config/dev.yaml
project: acme-ml-dev
location: europe-west4
staging_bucket: gs://acme-ml-dev-europe-west4-staging
pipeline_sa: trainer-churn@acme-ml-dev.iam.gserviceaccount.com
serve:
  machine_type: n1-standard-2
  min_replicas: 1
  max_replicas: 2

Define the infrastructure around the pipeline, such as buckets, service accounts, IAM bindings, Artifact Registry repositories and endpoints, in Terraform, and keep trained models and individual runs out of it. Terraform describes things that should exist continuously; a training run is an event. Mixing them produces plans that want to destroy a model.

Observability and debugging beyond the console

When something fails on this platform, the console shows you a state, and the state is rarely the cause. A job marked failed, an endpoint stuck deploying and a pipeline step in error are all the same message, and the real information is in Cloud Logging. A reliable debugging routine saves hours.

Start at the right layer. For a custom job, open the logs from the job page and read the worker's stdout and stderr: Python tracebacks, missing files and out-of-memory kills are all there. For a pipeline run, click the failed step and open its logs rather than reading the run-level error. For an endpoint deployment that fails, the container's startup logs usually show the crash or the health check path that never answered. For an online prediction error, the request and response logging you may have enabled, plus the endpoint's error metrics, tell you whether the container returned a 500 or the platform rejected the call before it arrived.

A short mapping from symptom to first action:

Query logs from the command line when you need to filter across many jobs. The exact log names vary by resource type, so find them once in the Logs Explorer and save the query:

gcloud logging read \
  'resource.type="aiplatform.googleapis.com/CustomJob" AND severity>=ERROR' \
  --project=acme-ml-dev --freshness=1d --limit=20 --format='value(timestamp,textPayload)'

Cloud Monitoring gives you endpoint metrics: request counts, latency, error rates, replica counts and CPU or accelerator utilisation. Alert on error rate and on latency percentiles, not on averages, which hide the slow tail that users feel. Replica count graphs are also the fastest way to see whether autoscaling is doing what you configured.

Model quality needs its own signal. Infrastructure being green says nothing about whether predictions are still correct. Model Monitoring can compare serving inputs to training data to detect skew and drift, and you should pair that with a business metric that arrives late, such as actual churn against predicted churn. Open-source libraries such as Evidently fill the same role if you prefer to run reports inside your own pipeline. Remember that Vertex Explainable AI is deprecated, with a shutdown in March 2027, so do not build new monitoring on it; the documented alternatives are libraries such as SHAP or LIME.

Cost, quotas and the Middle East

Cloud ML bills are made of a few repeating patterns, and knowing them is cheaper than a budget review after the fact.

Endpoints bill for every node-hour of every deployed replica, busy or not. This is the biggest line for most teams. Right-size the machine, keep the minimum honest, undeploy staging endpoints out of hours if nobody uses them overnight, and move anything that is really batch to batch prediction. Training bills per worker-hour while the job runs, so checkpointing plus cheaper preemptible capacity, where your job tolerates restarts, reduces cost, and a smaller machine with a longer run is not always cheaper. Pipelines add a small per-run charge on top of component compute, which matters only if you run thousands of tiny runs. Generative calls bill per token, so prompt length is a cost lever, and the context caching and batch options exist to reduce it; check the current pricing page rather than trusting a figure from a guide.

Labels turn this from guesswork into a report. Put team, model and env labels on jobs, models and endpoints (the examples above do), enable billing export to BigQuery and group by label. A budget alert at a modest threshold in every project catches the forgotten endpoint on day two instead of day thirty. For steady GPU needs, committed use discounts and reservations trade flexibility for price and for guaranteed capacity.

Quotas are the other constraint. Training GPUs, online prediction replicas and Gemini requests are limited per project and per region. A quota error is not a bug, so request an increase early and with a justification, because approval is not instant, and design for the quota you have, for example by falling back to another region only if data residency allows it.

For readers in the Gulf and Egypt, region choice is a compliance and latency decision before it is a technical one. Check the locations page for which Vertex AI features are available in the Middle East regions, such as the Dammam and Tel Aviv locations, because feature availability is region-specific and generative features often start in the global location or in the United States and Europe. If a regulation or a client contract requires data to stay in-country, confirm that every component of your design, training, endpoint, batch output, logs and model artifacts, can run in the permitted region before you build. It is far cheaper to learn that a feature is missing from a design document than from a failed deployment.

Putting it all together

Here is the whole loop for one real model, so you can see how the sections connect. A churn model for a telecom team should retrain monthly, be reviewed, roll out safely and stay cheap.

1. Infrastructure. Terraform creates a dev and a prod project's buckets named with their region, an Artifact Registry repository in that region, a trainer-churn and a serve-churn service account, and the IAM bindings, including iam.serviceAccountUser for the CI deployer.
2. Images. CI builds churn-train and churn-serve, tags them with the commit hash, and pushes them to Artifact Registry.
3. Pipeline. On merge, CI compiles the KFP pipeline, then submits it to dev as the training service account with a dated pipeline_root and versioned input paths. Steps: validate data, train with checkpointing, evaluate against the current champion, and register the model as a new version labelled with the commit, dataset and run, with the alias candidate.
4. Gate. If the candidate beats the champion on the held-out set, a reviewer approves, and the champion alias moves.
5. Rollout. A deploy step reads the champion alias and deploys that version to the prod endpoint at zero traffic on at least two replicas. It sends smoke-test traffic, shifts ten percent, watches error rate and latency in Cloud Monitoring, then shifts the rest. The previous deployment stays until the new one has proven itself, so rollback is a traffic change.
6. Batch. The nightly scoring job uses batch_predict against a BigQuery table and writes results back, so no endpoint idles.
7. Watch. Model Monitoring and a late-arriving business metric compare reality with predictions; budget alerts and label reports track cost.
8. Maintain. A pinned SDK, a CI test run that turns deprecation warnings into errors and a quarterly read of the deprecations page keep the upgrade surprise small.

None of these pieces is exotic, and that is the point. Reliability on this platform comes from conventions applied consistently: explicit identities, immutable artifacts, registry aliases, traffic splits, labels and pinned versions.

Where you are now

You can now reason about the platform instead of following tutorials. You know why resources are regional and what a resource name encodes. You can say which identity a job runs as and fix the actAs and image-pull failures. You can package training and serving as containers that honour the contract, build repeatable pipelines with correct caching, version models through registry aliases and release them with traffic splits and rollback. You can choose between online and batch prediction, track experiments, keep an SDK upgrade from breaking you, run the whole thing from CI without stored keys and attribute cost with labels.

Senior is the next part. It looks at the same platform as something you own for other teams: private networking with Private Service Connect, VPC Service Controls and customer-managed encryption keys, multi-tenant project design, quota and capacity planning, governance of who can promote what, incident playbooks, and the point at which another tool, such as SageMaker on another cloud or KServe on Kubernetes, is the better choice.

Sources

• Vertex AI documentation overview
• Vertex AI release notes
• Gemini Enterprise Agent Platform release notes
• Vertex AI deprecations
• Vertex AI Pipelines introduction
• Getting predictions overview
• Vertex AI access control
• Use the Vertex AI SDK for Python
• SDK migration guide for 2.x
• Agent Platform Python reference
• googleapis/python-aiplatform on GitHub
• google-cloud-aiplatform on PyPI

Part two of three. A cumulative review of Beginner and Mid-level material on Vertex AI, organised by topic rather than level, in about thirty-five minutes. Fast review first, scenario and trade-off questions after. Senior reviews all three.

One line of context for the interview itself: Google has been renaming the product to Gemini Enterprise Agent Platform. Interviewers still say Vertex AI, job posts use both, and the service name, the gcloud ai commands and the resource names have stayed familiar. Say "Vertex AI, now part of Gemini Enterprise Agent Platform" once and move on.

Where this picks up

Foundations, in one screen

Vertex AI is Google Cloud's managed platform for the machine-learning lifecycle. You bring data and code, it runs training on machines it provisions, stores versioned models in a registry, serves them from autoscaled endpoints or batch jobs, and orchestrates the whole chain with pipelines. Almost everything is regional, and everything runs as an identity you choose.

If you can say that in thirty seconds and then name the four nouns, project, region, model and endpoint, you have passed the first screen.

Questions you should answer fluently

Why does a job work from your laptop and fail in a pipeline? The identity changed. On a laptop the SDK uses your user credentials from gcloud auth application-default login; in a job it runs as the custom service account you gave it, or a default account if you gave none. The service account lacks a permission you have. Fix it by granting that account the specific role or bucket access it needs, never by running the job as yourself.

What are the two grants needed to submit a job as a service account? The service account needs the roles its code uses, and the submitter needs roles/iam.serviceAccountUser on it. Without the second, submission fails with an iam.serviceAccounts.actAs denial.

What does the Vertex AI service agent do? It is a Google-managed account the platform uses to act on your behalf, for example to pull your training image from Artifact Registry. It is different from your runtime service account. A failure to pull an image is fixed by giving the service agent read access to the registry, not by changing your own roles.

Why are regions such a big deal? Resources are regional, a model in one region cannot be served from an endpoint in another, and data residency follows the region. Buckets, images, jobs and endpoints should line up, and for regulated employers the region is a compliance decision. Feature availability also varies by region, so check before designing.

What is a pipeline's pipeline_root for? It is the Cloud Storage location where step outputs and artifacts are written. Small values pass between steps as parameters; files such as datasets and models pass as artifacts whose paths live under that root. It must be writable by the pipeline's service account.

What does pipeline caching key on? The step's inputs and its component definition, not the contents of the data behind a path. If you pass the same gs://.../latest/ string and the files change, the step is served from cache. Version the input path or disable caching for that step.

How do you release a model safely? Deploy the new version beside the current one at zero percent traffic, test it, shift a small share, watch error rate and latency, then shift the rest. Keep the previous deployment until the new one has proven itself, because rollback is then a traffic change in seconds.

Online or batch? If a user is waiting, online. If the results are consumed later in bulk, batch, because it releases its machines when done and an endpoint bills per node-hour even when idle.

Scenario questions

"Our training job fails within a minute and there are no training logs. What do you check?"

A strong answer works through layers in order. No training logs means your code probably never started. First, the image: can the Vertex AI service agent pull it from Artifact Registry? A permission error on pulling is the commonest cause, especially with a registry in another project. Second, the entrypoint: does the container start when you run it locally with the same arguments? Third, the job spec: machine type, quota and region. Cloud Logging for the job shows platform-level messages even when the container produced none.

A weak answer says "check the logs" without saying which, or restarts the job hoping for a different result.

"A deploy has been stuck for fifteen minutes and then fails with 'model server never became ready'."

The platform started your container and the health check never succeeded. Read the container's startup logs in Cloud Logging for a crash or a long model load. Then check the serving contract: the server must listen on the port in AIP_HTTP_PORT, bind to 0.0.0.0 rather than localhost, and answer AIP_HEALTH_ROUTE with success. Reproduce locally by running the same image with those environment variables set and calling the health and predict routes with curl. If the model is large, check that it loads before the health route reports ready, and that the machine has enough memory.

A strong answer also mentions the cheap way to avoid this entirely: test the image locally before deploying.

"Predictions are intermittently returning 503 during traffic spikes."

New replicas need time to pull the image, start and load the model, so capacity arrives after the spike does. Check the replica count graph in monitoring against request rate. Mitigations: raise the minimum replicas to cover normal peaks, shorten model load time, check that the maximum replicas is not blocked by quota, and make the client retry with exponential backoff and jitter. Retrying immediately in a tight loop amplifies the problem. If the traffic is bursty and latency is relaxed, consider whether it should be batch.

"The bill doubled this month. Where do you look?"

Start with billing export grouped by label and by SKU. The usual suspects are idle endpoints with GPU replicas, a raised min_replica_count, training jobs on larger machines than needed, and a forgotten experiment. Without labels you are guessing, which is the lesson: label jobs, models and endpoints with team, model and environment from day one. Then act: undeploy and delete what nobody uses, move non-interactive work to batch, add budget alerts so the next one is caught early.

"A pipeline step did not rerun although the source data changed."

Caching. The step's cache key did not change because the parameter was the same string, so the previous output was reused. The data behind the path is not part of the key. Fix by passing versioned paths, such as a dated folder, or by disabling caching for that step. A strong answer adds that caching is still valuable, and the fix is to make inputs explicit rather than to turn it off everywhere.

"After upgrading the SDK, from vertexai.generative_models import GenerativeModel fails."

The five generative modules, generative_models, language_models, vision_models, tuning and caching, were deprecated in June 2025 and scheduled for removal in June 2026. New code uses the Google Gen AI SDK: from google import genai, then genai.Client(vertexai=True, project=..., location=...). The deeper answer is about process: pin the SDK version, run tests with deprecation warnings as errors, and read the release notes before upgrading.

"A model's quality dropped but all dashboards are green."

Infrastructure metrics say the service is up, not that predictions are right. Check for input skew or drift with Model Monitoring or a drift report, look at the business metric that arrives late, and check whether an upstream data change landed. Then use the registry labels on the live version to find which data and code produced it, and compare with the previous version. If the previous version was better on the same held-out data, roll back through the traffic split while you investigate.

Trade-off questions

Vertex AI Pipelines or Airflow?

Pipelines is serverless, native to Google Cloud, tracks lineage of models and artifacts, and uses the KFP SDK, so there is no cluster to run. Airflow is a general scheduler with a large operator ecosystem, and fits when ML is one workload among many orchestrated tasks, or when you are not committed to one cloud. A common split is Airflow for data engineering and Vertex AI Pipelines for the ML steps, triggered from Airflow. Say what you would choose for a given team and why.

Vertex AI Experiments or MLflow?

Experiments needs no server and inherits Google Cloud IAM and project boundaries. MLflow has a broader open-source ecosystem and a richer comparison interface, and travels across clouds. The wrong answer is running both as competing sources of truth. Pick one record of truth and link model versions back to it with labels.

Prebuilt container or custom container?

Prebuilt for standard frameworks with standard requirements; less to maintain, and the serving contract is handled. Custom when you need system packages, custom pre- or post-processing, an unsupported framework or a build you must reproduce exactly. Custom containers cost maintenance and security patching, so the justification should be specific.

One project or several?

Several. Separate dev, staging and prod projects give you a hard boundary for IAM, quotas, billing and later network controls. One shared project is faster to start and harder to secure and to bill. A middle path is separate projects per environment and labels per team within each.

Endpoint with a high minimum or batch?

An endpoint is for latency; batch is for cost on non-interactive loads. If the data arrives hourly and consumers read it later, batch wins by a wide margin. If a checkout page needs a score in fifty milliseconds, you need an endpoint, with at least two replicas.

Spot capacity or on-demand?

Spot or preemptible capacity is cheaper and can be reclaimed at any time. It is appropriate when training checkpoints regularly and resumes, and inappropriate for an endpoint serving users or for a job without checkpoints. State the condition, not just the saving.

Pin the SDK, or follow latest?

Pin, because the package releases roughly weekly and a 2026 major version moved namespaces and removed modules. Upgrade deliberately, in a branch, with deprecation warnings as errors. The cost of pinning is a small recurring chore; the cost of not pinning is an unannounced outage.

What a strong answer covers that a weak one misses

The pattern behind these is simple. A mid-level engineer is trusted because their answers include the failure mode and the operational consequence, not just the happy path.

Commands and calls to have ready

You will not be asked to recite flags, but you should be able to sketch these without looking anything up:

gcloud auth application-default login
gcloud config set project PROJECT_ID
gcloud services enable aiplatform.googleapis.com
gcloud ai models list --region=us-central1
gcloud ai endpoints list --region=us-central1
gcloud ai custom-jobs list --region=us-central1
gcloud ai endpoints deploy-model ENDPOINT_ID --region=us-central1 \
  --model=MODEL_ID --display-name=DEPLOYMENT_NAME \
  --machine-type=n1-standard-4 --min-replica-count=2 --max-replica-count=6

aiplatform.init(project=P, location=L, staging_bucket="gs://...")
job = aiplatform.CustomContainerTrainingJob(display_name="t", container_uri=IMAGE,
        model_serving_container_image_uri=SERVE_IMAGE)
model = job.run(replica_count=1, machine_type="n1-standard-8", service_account=SA)
endpoint = aiplatform.Endpoint.create(display_name="e")
model.deploy(endpoint=endpoint, machine_type="n1-standard-4",
             min_replica_count=2, max_replica_count=6, traffic_percentage=0)
endpoint.undeploy_all(); endpoint.delete()

If an interviewer asks you to write a pipeline, the shape is: a @dsl.component per step, a @dsl.pipeline with a pipeline_root, compiler.Compiler().compile(...), then aiplatform.PipelineJob(...).submit(service_account=...). Say aloud that the service account is explicit and that the input paths are versioned.

Questions to ask them

Good candidates ask questions too, and these show you think about operations:

• Which identity do your pipelines run as, and who can act as it?
• How do you promote a model from dev to prod, and who approves it?
• What region constraints apply to your data, and are there features you could not use because of them?
• How do you attribute cost to teams, and how often has an idle endpoint surprised you?
• How do you handle SDK and model-version retirements?

Common mistakes in interviews

The first is naming a feature as if it were permanent. This platform retires things on dated schedules: legacy notebooks, AutoML text and video, the legacy Feature Store and Vertex Explainable AI all have dates in the deprecations page. If you recommend one of them, say you checked it is still supported. The second is giving only the console path: "I click Deploy". Interviewers want the SDK or Terraform route, because the console is not repeatable. The third is ignoring cost: a design with a GPU endpoint and no mention of what it costs when idle reads as inexperienced. The fourth is forgetting that Gemini model IDs are retired, so a candidate who hard-codes one invites the question of what happens next quarter.

A fifth, quieter mistake is treating pipelines as a notebook with extra steps. The reasons to use them are repeatability, caching, lineage and unattended scheduling; if you cannot say which of those your design needs, you have not justified the pipeline.

Sixty-second self-test

Answer each aloud in one or two sentences. Anything you hesitate on is your study list.

1. Which identity does a pipeline step run as, and which two grants are needed to submit as it?
2. Why does a bucket's location matter relative to your job?
3. What is pipeline_root, and what goes under it?
4. What exactly does the pipeline cache key include, and what does it miss?
5. What does min_replica_count cost you even with no traffic?
6. How do you roll out a new model version and how do you roll back?
7. What are the four AIP_ variables, and what must a serving container do with them?
8. When is batch prediction the right choice?
9. Which import replaces vertexai.generative_models, and why must the model ID not be a literal?
10. Where do you look first when a job fails with no training output?
11. How do you attribute cost to a team?
12. What is the first thing you do at the end of a lab?

If you can answer eleven of the twelve without hesitation, you are ready for a mid-level screen on this platform. Senior takes each topic further, into private networking, perimeters and customer-managed keys, multi-tenant design, capacity planning, governance of promotion, incident response, and the question of when to choose something other than Vertex AI. For neighbouring topics, see the guides on Kubeflow, Docker, Terraform and Evidently.

Part two of three. At this level your jobs run and your models deploy, so the problems change character. They stop being "why will this not start" and become "why does this work on my laptop and fail in the pipeline", "why did the bill jump" and "why did the library upgrade break production". Start with the error table, then the practices in order of leverage, then the practice cards and recipes.

Common errors at this level

Cumulative: Beginner's errors still apply, and these appear once things mostly work. Exact message wording changes between library and service versions, so treat the quoted fragments as things to search for, and confirm against your own logs.

Practices that pay off, in order of leverage

The first three change your life; the rest compound.

1. Give every workload its own service account

One service account per pipeline or service, with roles scoped to the specific buckets and secrets it needs, removes the largest class of mid-level incidents: the permission that works on your laptop and nowhere else, and the over-broad account that turns a leaked token into a project-wide breach. Name accounts after the workload, such as trainer-churn, so audit logs read like sentences.

It also makes debugging quicker. When a 403 names trainer-churn@..., you know exactly which grant to add, and nobody is tempted to widen a shared account "just to get unblocked".

2. Make artifacts immutable and traceable

Image tags that include the commit, dated data folders, and labels on every model version that record the commit, dataset version and pipeline run. The test is whether you can answer "what produced the model serving traffic today" in two minutes, from the version's labels alone. If the answer needs a person's memory, the process has a gap.

3. Release with traffic splits, never in place

Deploy the new version at zero percent, test it, shift a small share, watch, then complete. Keep the old deployment running until you trust the new one. The rollback is a traffic update, which takes seconds, instead of a redeploy, which takes many minutes and may fail on the same capacity problem you are trying to recover from.

4. Label everything and export billing

Labels team, model and env on jobs, models and endpoints, plus billing export to BigQuery, convert a mystery bill into a query. Add a budget alert per project at an amount you would notice. This costs an hour once and has paid for itself in every team that has done it.

5. Pin the SDK and fail CI on deprecations

The package ships often and a major release in 2026 moved modules. Pin exact versions per entry point, upgrade on purpose, and run tests with python -W error::DeprecationWarning so removed APIs show up in a pull request. Read the deprecations page quarterly: it lists dated shutdowns for the legacy Feature Store, Vertex Explainable AI and older model versions.

6. Test the serving image locally before deploying

Running the image with the AIP_ variables set, then calling health and predict with curl, catches most deploy failures in seconds. A failed deploy costs a quarter hour of waiting and teaches you little; a local failure costs a few seconds and shows the traceback.

7. Put the model ID, region and machine sizes in configuration

A literal model name in code is a retirement waiting to happen. One config file per environment, read by the submission script, also means dev and prod differ in exactly one place, which makes "why does prod behave differently" answerable.

8. Checkpoint every long job

Write checkpoints to Cloud Storage and resume from the latest on start. It makes spot capacity usable, makes preemption a nuisance rather than a loss, and lets you cancel and resume a job when you change your mind about the machine type.

9. Prefer batch for anything not interactive

Move nightly scoring and bulk back-fills off endpoints. You stop paying for idle replicas, you avoid cold-start failures, and the output lands in a table analysts can query.

10. Use workload identity federation in CI

No downloaded key files, no secrets to rotate, and CI jobs get short-lived access as a named service account. For GitHub Actions, the google-github-actions/auth action does the exchange.

Habits in the notebook, the pipeline and the console

A few small habits matter disproportionately.

Print the identity and project at the top of every script that touches the platform. A line that shows the credentials type, service account email and project catches wrong-project mistakes in the first second instead of after a long job. Pair it with an assertion that the staging bucket's location matches your region.

Do not train in the notebook that you used to explore. A notebook holds hidden state, such as variables defined out of order and cells run twice. Move training into a script or a pipeline component with explicit inputs, and keep the notebook for exploring and plotting. The same code then runs identically in CI.

Treat the console as a viewer. Create resources through code or Terraform, and use the console to look at them. Anything created by clicking cannot be reviewed, repeated or rolled back, and eventually someone will click the wrong project.

Keep notebooks on Workbench instances. Older managed and user-managed notebook products reached end of support and deletion, so teams still depending on them should have migrated by now. Workbench instances are the supported route, and they get scheduled upgrades.

Practice cards

Each card is a ten-to-thirty minute exercise on a sandbox project. Do them with a budget alert in place and delete what you create.

Operational recipes

Short, repeatable procedures to keep in your team's runbook.

Recipe: find out what is running and costing money

for r in us-central1 europe-west4 me-central2; do
  echo "== $r =="
  gcloud ai endpoints list --region=$r --format='table(name.basename(),displayName,deployedModels.len())'
  gcloud ai custom-jobs list --region=$r --filter='state=JOB_STATE_RUNNING' \
    --format='table(name.basename(),displayName,createTime)'
done

Adjust the region list to the regions your organisation uses; running the loop over a region where you have nothing simply returns empty tables. Anything that appears and has no owner label is a candidate for a conversation. Schedule this weekly and post the output to a team channel.

Recipe: tear down an endpoint completely

from google.cloud import aiplatform

aiplatform.init(project="acme-ml-dev", location="europe-west4")
endpoint = aiplatform.Endpoint("ENDPOINT_ID")
endpoint.undeploy_all()
endpoint.delete()

Undeploy first, because an endpoint with deployed models cannot be deleted without it. Afterwards, check that the model itself is still in the registry, which is what you want, and that no batch jobs were pointing at the endpoint.

Recipe: roll back a bad release

endpoint = aiplatform.Endpoint("ENDPOINT_ID")
for m in endpoint.list_models():
    print(m.id, m.display_name, m.model, m.model_version_id)

endpoint.update(traffic_split={"PREVIOUS_DEPLOYED_ID": 100, "BAD_DEPLOYED_ID": 0})

First list the deployed models to get their IDs, since the split is keyed by deployed model ID. Move all traffic back, confirm error rates recover in Cloud Monitoring, and only then undeploy the bad version. Write down what you saw before you tidy up, because the logs and metrics of the failed version are your evidence.

Recipe: find the errors in a failed job

gcloud ai custom-jobs describe JOB_ID --region=europe-west4 \
  --format='value(state,error.message)'

gcloud logging read \
  'resource.type="aiplatform.googleapis.com/CustomJob" AND severity>=ERROR' \
  --project=acme-ml-dev --freshness=2d --limit=30 \
  --format='value(timestamp,textPayload)'

The first command gives you the platform's summary, which is often vague. The second gives you the worker's real error. If both are empty, the container probably never started, so check the image pull permissions for the service agent.

Recipe: confirm which identity code uses

import google.auth
creds, project = google.auth.default()
print(type(creds).__name__, getattr(creds, "service_account_email", "user"), project)

Run it at the top of a script locally and inside a job, and compare. It resolves most "works here, fails there" questions in one line.

Recipe: a weekly hygiene checklist

• Read the Vertex AI release notes and the deprecations page for anything that touches what you use.
• Run the inventory loop above and tag or delete orphans.
• Check budget alerts and the top three label groups in the billing export.
• Check quota usage for GPUs and online prediction in your busiest region.
• Confirm that the pinned SDK is still within the supported Python versions.

Recipe: a pre-deploy checklist for a model version

• Labels on the version include commit, dataset and pipeline run.
• The serving image ran locally with the AIP_ variables and both routes answered.
• The candidate beat the champion on the same held-out data.
• The new deployment starts at zero percent traffic with at least two replicas.
• Alerts exist for error rate and latency percentiles on the endpoint.
• You know the rollback command and have its deployed model IDs written down.

Habits for working in the Middle East

A few considerations apply specifically to teams in the Gulf and Egypt, and are easy to forget until they bite.

Decide the region from policy, not convenience. Many tutorials use us-central1, and copying that default into a regulated workload is how a compliance review fails. Write down which regions each environment may use, check the locations page for the features you need in them, and put the region in your bucket names so it is visible. Generative features often start in global or in US and EU regions, so confirm availability for the region you need, and if it is not there, take that to your compliance contact before building.

Plan for quota with lead time. GPU quota in a smaller region can be thin, and increases take time to approve. Ask when you start designing, not the day before a launch.

Expect latency from your users to the region you choose, and measure it. A region close to your users reduces round-trip time for online prediction, but batch work is tolerant, and you may run it where capacity is available if policy allows.

Finally, document your assumptions. When an auditor or a client asks where data lives and who can reach it, you want a one-page answer that names the project, the region, the service accounts and the encryption configuration, not an afternoon of console archaeology.

A short checklist

Before you call a Vertex AI workload production-ready, check that:

• It runs as a dedicated service account with scoped roles.
• Images and data paths are immutable and versioned.
• Model versions carry labels and are promoted by alias.
• Rollout uses a traffic split and rollback was practised.
• The endpoint has at least two replicas, or you chose batch instead.
• Jobs and endpoints are labelled and a budget alert exists.
• The SDK is pinned and CI treats deprecation warnings as errors.
• No secret appears in code, images, pipeline YAML or parameters.
• The region was chosen by policy and appears in resource names.
• There is a written teardown for every temporary resource.

Senior takes these practices from "my project" to "a platform other teams rely on": private networking, perimeters and encryption keys, quota and capacity planning across teams, promotion governance, incident playbooks and the decision about when this is not the right tool. For neighbouring topics, see MLflow, Docker and Terraform.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when Vertex AI is a platform that other teams build on, rather than a tool you use yourself.

Up to now the questions were about making a training job run and a model answer requests. From here they change: which identity is a job actually running as, what happens to forty teams when one of them exhausts a regional GPU quota, what will your bill look like when someone forgets an endpoint, and how you absorb a platform whose own SDK changes shape in a major release.

Where this picks up

I am starting with the naming, because it is the first thing a platform owner has to explain to everyone else.

One platform, two names

In 2026 Google began folding "Vertex AI" into a product it now calls Gemini Enterprise Agent Platform. The official documentation still lives under the vertex-ai path and redirects between the two, and release notes are published under both. You will meet both names in vendor decks, job posts, tutorials and your own colleagues' messages, and part of owning the platform is making sure nobody concludes that two products exist.

What matters for a platform owner is what did and did not move. The classic machine-learning surface that your teams depend on, meaning custom training, the Model Registry, endpoints, batch prediction and pipelines, kept its resource model: the paths still look like projects/P/locations/L/endpoints/E, the gcloud ai command group is where you left it, and the API is still aiplatform.googleapis.com. Treat the name change as a documentation and console change, and re-verify those three anchors against the current docs whenever you write them into a standard, because that is the kind of detail a rebrand can quietly change later.

What did move is the Python SDK, and that is where your teams will feel pain. Version 2 of google-cloud-aiplatform (2.3.0 is current at the time of writing, with releases arriving roughly weekly) is a breaking major release. The generative modules were migrated to the Google Gen AI SDK, google-genai. The agent surface was split into its own distribution, google-cloud-agentplatform, imported as agentplatform. Classic ML stays in google-cloud-aiplatform. I come back to what this means for upgrades in its own section; for now, hold the idea that the platform's concepts are stable while its client libraries are not, and design your own standards around the stable layer.

Architecture and failure modes

Vertex AI is a control plane in front of compute you rent by the hour. You submit a request (a training job, a pipeline run, a deployment), the control plane schedules it onto machines in the region you named, and the machines run your container as an identity you chose. Almost every failure you will own sits at one of the seams in that sentence.

Walk the seams in order and name what breaks at each.

Submission. The caller needs permission on the Vertex AI API and, if the job runs as a custom service account, permission to act as that account (the iam.serviceAccounts.actAs permission, usually granted through roles/iam.serviceAccountUser on that one account). The classic failure is a pipeline that submits fine from a laptop and fails from CI, because the human has broad rights and the CI identity does not. The fix is never to widen the CI identity to project owner. It is to grant the one actAs it needs.

Scheduling. A job in a region needs quota in that region for the accelerator type it asks for. When quota is missing the job fails or waits, and no amount of retrying changes it. This is the most common reason a platform "goes down" for a team on a Monday: not an outage, but a quota that was never raised. I treat it separately in the scaling section.

Image pull and startup. The job's machines pull your training or serving image from Artifact Registry. If the Vertex AI service agent cannot read that repository, the job dies before your code runs. The same is true for a serving container that starts but never answers its health route: the deployment fails because the model server never became ready, and the evidence is in Cloud Logging rather than in the error returned to the caller. Platform owners who keep a single, boring, documented serving-container contract (an HTTP server, a health route, a predict route, and the AIP_* environment variables Vertex supplies) have far fewer of these tickets than those who allow every team to invent their own.

Run time. The job runs as the identity you attached. Whatever that identity can read, your code, and anyone who can influence your code, can read. This is the seam the security section is about.

Registration and deployment. The model lands in the registry as a version; a deployment places that version on replicas. A deployment is where cost begins to accrue continuously, so it is where leaks of money happen.

Serving. Replicas scale between a minimum and a maximum. Scale-up is not instant: new replicas must be provisioned, pull the image, load the model and pass the health check, and during that window the existing replicas absorb the load or requests fail. A model that takes ninety seconds to load will not help you through a sudden spike, regardless of what the autoscaling target says.

Notice that none of these failures is a Vertex AI bug in the usual sense. They are the platform behaving as designed around a permission, quota or contract that nobody owned. A large part of the senior role is to make each of those things owned.

Scaling: which component breaks first

On a self-hosted stack the first thing to break is usually a database or a scheduler. On a managed platform it is almost always a limit someone else set: a quota, a regional capacity shortage, or a rate limit. Plan for those before you plan for anything you can tune.

GPU and accelerator quota is the first wall. Quotas are per project, per region and per accelerator type. A team that trained happily on T4s in development discovers in production that the larger accelerator they planned to use has a quota of zero in the region their data must stay in. Requests for increases are reviewed by people and take time, so the habit is to file them as part of capacity planning, weeks before launch, with a realistic justification. If you wait for the error, you have already lost the week.

Regional capacity is the second wall. Even with quota, accelerators in a given region can be scarce at the moment you ask. For predictable demand, reservations are the mechanism: you commit to capacity ahead of time rather than hoping it is free. For spiky demand, design the job to be restartable and to checkpoint, so a spot or preemptible-style interruption costs you minutes instead of the run. Whether a given discount model is available for a given machine type in a given region is a docs-and-pricing question; look it up per case, do not assume.

Online endpoints are the third. Each endpoint scales between a minimum and maximum replica count driven by utilization targets such as CPU or accelerator duty cycle. Three things go wrong at scale:

1. The maximum is set below real peak, so latency climbs and requests fail while the platform is "working as configured".
2. The minimum is one, so a single replica failure or rolling update takes the model to zero capacity. For anything with an availability promise, run at least two.
3. Cold scale-up is slower than the traffic ramp. Model load time, not machine provisioning, is often the dominant term, so measure it and decide whether to keep headroom instead.

Request size is a limit too. Online prediction has a payload ceiling (around 1.5 MB at the time of the fact-check; read the current limits page before quoting a number to a team), and the answer to "my payload is too big" is not to chunk it through an endpoint but to move that workload to batch prediction or to pass references to data in Cloud Storage or BigQuery.

Batch prediction is your scaling lever for cost. It has no idle endpoint, it reads from and writes to Cloud Storage or BigQuery, and it uses machines only for the duration of the job. The platform-owner question for every new "real-time" request is whether it needs to be real-time. A nightly scoring job that was built as an always-on endpoint is the most common avoidable cost in the whole platform.

Pipelines scale by fan-out, and fan-out hits quota. A pipeline that launches a hundred parallel training components asks for a hundred components' worth of compute at once. Give pipeline steps explicit machine types, and put a concurrency limit in your pipeline templates where the design allows one, so a single run cannot consume the project's whole quota and starve everyone else's.

Generative traffic has its own limits. Calls to hosted Gemini and partner models are governed by per-model, per-region quotas and a shared-capacity behaviour, and the platform offers Provisioned Throughput for committed capacity. Treat a 429 from a model call as a capacity signal: the client needs retries with exponential backoff and jitter, and the platform needs a decision about whether a given workload deserves committed throughput. Model identifiers change quickly, and old ones are retired on published dates (Gemini 1.5 access ended on 24 September 2025, for instance), so keep model names in configuration, never in code.

Security: the trust model

Start from one sentence: a Vertex AI job is arbitrary code running as a service account inside your project. Everything in security follows from taking that literally.

Two kinds of identity

Vertex AI uses Google-managed service agents, accounts the platform itself uses to do work on your behalf, such as pulling an image or writing logs. You do not control their code, but you do control what they are granted. Separately there is the custom service account you attach to a job, pipeline or endpoint. This is the identity your code runs as, and the one that reaches your data.

The default, if you attach nothing, is a broadly privileged account, and that is the mistake to design out. The platform standard is: every pipeline and every job type gets its own dedicated service account with the least privilege it needs, created through Terraform, with no keys. Users do not run production jobs under their own credentials, and nothing in production uses a downloaded service-account key.

# A dedicated runner identity for one pipeline, with no key ever created
gcloud iam service-accounts create churn-train \
  --project=ml-churn-prod \
  --display-name="Churn training pipeline"

# The pipeline's identity may use Vertex AI and read exactly one bucket
gcloud projects add-iam-policy-binding ml-churn-prod \
  --member="serviceAccount:churn-train@ml-churn-prod.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

gcloud storage buckets add-iam-policy-binding gs://ml-churn-prod-features \
  --member="serviceAccount:churn-train@ml-churn-prod.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

# The CI identity may act as that account, and only that account
gcloud iam service-accounts add-iam-policy-binding \
  churn-train@ml-churn-prod.iam.gserviceaccount.com \
  --member="serviceAccount:ci-deployer@ml-platform.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

Role names such as roles/aiplatform.user should be checked against the current IAM reference when you write them into policy, since the product is mid-rename; the pattern, a narrow role on the project plus a narrow data role on exactly the resources needed, is what matters.

Who can act as whom

The actAs permission is the lever attackers and accidents use. Whoever can submit a job as service account X can run any code with X's permissions. So roles/iam.serviceAccountUser is not a harmless convenience role: granting it on a highly privileged account hands that account's power to everyone who holds it. Audit it as you would audit a grant of the privileged role itself. Prefer granting it on individual service accounts, never at project level, and never to a broad group.

Network boundaries

Three layers matter, and a mature platform uses all of them for sensitive workloads.

Private endpoints. Online endpoints can be public (the shared type, or the dedicated public endpoint the docs now recommend over it) or private through Private Service Connect, which the docs recommend for private serving. If a model scores sensitive data from inside your network, the endpoint should not have a path from the internet at all. Older VPC-peering private endpoints exist, but confirm their current status in the docs before building a new design on them.

VPC Service Controls. A service perimeter around the projects and APIs involved is the control that stops data leaving even when an identity is compromised: a stolen token used from outside the perimeter fails, and a job inside the perimeter cannot copy a dataset to a bucket in another organization. Expect a period of friction when you first turn a perimeter on, because legitimate flows (a vendor's tool, a public package mirror, a cross-project model deployment) will be blocked and need explicit ingress or egress rules. Roll the perimeter out in dry-run mode first, read the violations it logs, then enforce.

Job networking. Custom jobs can run attached to your VPC through the network setting, so training traffic reaches private data sources and egress is governed by your firewall and proxy rules. For regulated data, make that the default in your job templates rather than something each team must remember.

Encryption and audit

Customer-managed encryption keys are configured through the encryption_spec_key_name setting on resources that support it, and can be made a default at the aiplatform.init() level for SDK-driven work. Treat CMEK as a decision with operational weight: if the key is disabled or destroyed, the data it protects becomes unreadable, so key ownership, rotation and break-glass procedures have to be written before the first resource uses it. Turn on Data Access audit logs for the aiplatform.googleapis.com service in the projects that matter, so that "who ran what as whom" is answerable from logs rather than from memory.

Secrets and the supply chain

A job's inputs are code, container images, Python packages, pipeline definitions, data and secrets. Each is a way for something you did not write to run with your identity.

Secrets. Pipelines can reference Secret Manager secrets, so credentials for a database or a third-party API are fetched at run time under the pipeline's service account rather than pasted into a component. Hold the same line you would hold anywhere: no secrets in container images, no secrets in pipeline YAML, no secrets in job arguments (which are logged and visible in the console), and no long-lived service-account keys. For CI, use Workload Identity Federation so a GitHub Actions or similar runner exchanges its own short-lived token for a Google identity; the google-github-actions/auth action implements it. See GitHub Actions for the pipeline side and Vault if your secrets live outside Google.

Images. Every custom training and serving container should come from your own Artifact Registry repository, built in CI, pinned by digest in the job definition rather than by a floating tag, and scanned before it is used (see Trivy). Do not point new work at the old gcr.io registry; Artifact Registry is the maintained home, and confirm the current guidance before you migrate existing references. Google's prebuilt training and prediction images are convenient, but they are tagged by framework version and updated over time, so pin a specific tag and treat a tag bump as a reviewed change.

Packages. A KFP component declared with packages_to_install runs pip install when the component starts. That is a supply-chain hole twice over: it installs whatever the index returns that day, and it does so at every run. For anything beyond prototypes, bake dependencies into an image you built and scanned, pin versions, and let components use that image as their base_image.

Pipeline definitions. A compiled pipeline is code that will run with the pipeline's identity. Keep the source in version control with required review, compile it in CI, store the compiled artifact in a location only CI can write to, and have the pipeline's submitter run only that artifact. The risk this removes is a person submitting a hand-edited YAML that nobody reviewed.

Models as artifacts. A serialized model file can execute code when it is loaded, which is true of several common formats. Only load model artifacts from locations your own pipelines write to, and be cautious when importing models from outside through Model Garden or other catalogs: review the provenance and licence, and put third-party models through the same registry and approval path as your own.

Multi-tenancy

The real question when several teams share Vertex AI is where the isolation boundary sits. There are three honest options, and they trade isolation against overhead.

My default recommendation is the first. Quota, billing and IAM all attach to the project, and those are exactly the three things teams fight over. Project creation should be a Terraform module or a self-service workflow, not a ticket, otherwise teams will crowd into shared projects to avoid the wait and you will have undone the isolation by accident. See Terraform for the module pattern.

Within a shared project the controls are weaker but real: a service account per team and per workload, labels on every job, endpoint and model for attribution, IAM conditions where they apply, and a naming convention enforced in the templates. Be honest with teams about what this does not give them: it does not give each team its own quota, and it does not stop a team with project-level rights from seeing everyone's resources.

Multiple models can be deployed to one endpoint, and the registry supports many models per project, but a shared endpoint shares its blast radius: a bad deployment, a memory leak or a traffic spike in one model affects the others on it. For models owned by different teams, give each its own endpoint, and share only things that are genuinely common, such as a base serving image.

Two further tenancy concerns are easy to miss. First, data: the strongest isolation is not at the Vertex layer but at the BigQuery datasets and Cloud Storage buckets the jobs read, so make per-team access to data part of the same Terraform module as the project. Second, a central registry or a per-team one: a central platform project holding approved base images and shared pipeline components makes consistency easy, and a per-team registry makes ownership clear. A common compromise is a central, read-only catalogue of images and components, and per-team model registries.

Cost

Vertex AI is billed in node-hours, worker-hours and tokens, and almost every surprise falls into a short list.

Idle endpoints. An online endpoint bills for its replicas whether or not anyone calls it. Experiments leave behind a deployed model, the team moves on, and the bill continues indefinitely. This is the highest-frequency cause of an unexpected invoice. The control is detection: a scheduled job that lists endpoints and their last-request time across all projects, labels the owner, and notifies, then escalates to undeploying anything unowned. Teardown is endpoint.undeploy_all() followed by endpoint.delete().

Minimum replica counts. Setting min_replica_count to two for availability doubles the baseline cost of an endpoint. That is often the right trade, and it should be a conscious one with a named owner, not a default copied from a tutorial. For low-traffic models, ask whether batch prediction, or a smaller machine type, serves the need.

Over-provisioned machines. Teams choose a large machine type for a first run and never revisit it. Endpoint metrics for CPU and accelerator utilization tell you whether the hardware is justified; sustained low utilization is a right-sizing ticket.

Training waste. Failed runs that retry for hours, hyperparameter sweeps with no stopping rule, and pipelines with caching disabled all burn compute. Execution caching in pipelines avoids re-running steps whose inputs did not change, and should be on unless a step is deliberately non-deterministic. Give long jobs timeouts, and give sweeps a maximum trial count. Where interruptions are tolerable, discounted capacity (spot VMs) reduces cost; confirm the current options and constraints for your machine type before promising a number.

Token spend. Generative workloads bill per token. Context caching and batch modes exist to reduce this, and prompt length is a cost lever teams rarely track. Put per-application budgets on generative traffic and alert on them.

Attribution. None of the above is manageable without knowing whose it is. Make labels mandatory in your templates (team, service, env, cost-center), export billing data to BigQuery, and report cost per label. Set budgets with alerts per project; project-per-team makes this trivial and a shared project makes it painful. Where GPU use is steady and predictable, committed-use discounts and reservations lower the price in exchange for a commitment, which only makes sense once you have a few months of utilization data.

Upgrades and migrations

This is where Vertex AI differs most from a tool you run yourself. You do not upgrade a server; you upgrade client libraries, container images and your own code against a service that changes underneath you, and you absorb deprecations that Google dates.

The SDK

The google-cloud-aiplatform package releases about weekly. Version 2.0.1 was the breaking major, and its effects show up in four ways teams commonly meet:

• Code that imports vertexai.generative_models, vertexai.language_models, vertexai.vision_models, vertexai.tuning or vertexai.caching was deprecated in June 2025 and scheduled for removal in June 2026; the replacement is google-genai (from google import genai). Treat any remaining import of those modules as already broken, and confirm against the current release what happens when you try.
• vertexai.Client is deprecated in favour of agentplatform.Client.
• Agent code moved: agent_engines became runtimes, sessions and sandboxes became top-level modules, and Memory Bank methods moved under client.memory_banks.
• Agent frameworks no longer take project and location from aiplatform.init(); configuration comes from the environment instead.

The platform answer is a short list of rules. Pin exact versions in a lock file for every service and pipeline image; do not let a weekly release reach production by accident. Turn deprecation warnings into errors in CI, so you find out in a pull request instead of during an incident. Use a single upgrade cadence: a monthly pull request that bumps the pinned SDK, runs the integration tests against a sandbox project, and reads the release notes between versions. And keep a small compatibility test per major use case (train, register, deploy, predict, run a pipeline) so that the question "does the new version still do the thing" has an automated answer.

# Fail CI on deprecation warnings so a removal never surprises production
import warnings
warnings.simplefilter("error", DeprecationWarning)

from google.cloud import aiplatform
aiplatform.init(project="ml-sandbox", location="us-central1")

Python itself is part of this: the SDK requires Python 3.10 or later, and older interpreters are stuck on frozen, older SDK versions. A team on an old Python is a team that cannot receive fixes.

Dated deprecations

The deprecations page is a calendar, and the platform owner's job is to put its dates in a tracker with an owner. As of this writing, items that matter include Vertex Explainable AI (deprecated 16 March 2026, shutdown 16 March 2027, with SHAP and LIME as the suggested alternatives), the legacy Feature Store (deprecated 17 February 2026, shutdown 17 February 2027, moving to Feature Store V2 backed by BigQuery, with Optimized online serving on the same dates and Bigtable online serving or Vector Search as the move), the older managed and user-managed Notebooks (replaced by Workbench instances), and the shutdowns of AutoML Text and AutoML Video. Partner and Gemini model versions also retire on published dates, which is why model identifiers belong in configuration. Read the page quarterly; do not rely on remembering it. See Feast for the open-source feature-store option if you decide not to follow Google's path.

Migrations you will actually run

Most migrations here are in-place client changes (SDK, import paths) plus moves between managed services (legacy to new Feature Store). Handle both with the same shape: write the target in a branch, run both versions in parallel against shadow traffic or a copy of the data, compare outputs, then cut over with a rollback path. For endpoints, the traffic-split mechanism is the rollback path: deploy the new model version to the same endpoint at a small percentage, watch error rate and latency, and move traffic in steps.

from google.cloud import aiplatform

endpoint = aiplatform.Endpoint("projects/NUM/locations/us-central1/endpoints/ENDPOINT_ID")
model = aiplatform.Model("projects/NUM/locations/us-central1/models/MODEL_ID@7")

# Canary: ten percent of traffic to the new version, ninety stays where it is
model.deploy(
    endpoint=endpoint,
    machine_type="n1-standard-4",
    min_replica_count=2,
    max_replica_count=6,
    traffic_percentage=10,
)

Governance and promotion

Governance is the set of rules that make the safe path the easy path. For a Vertex AI platform it comes down to four mechanisms.

The registry as the release contract. A model version is a candidate until something promotes it. Aliases are the promotion mechanism: candidate for a version that passed training, a name like champion or production for the one serving. Deployment tooling should read the alias rather than a version number, and only the promotion pipeline, running as a dedicated identity, should be able to move a production alias. People should not. Record, on the version, the labels that make it auditable: the git commit, the pipeline run, the dataset version and the evaluation report location. Lineage in ML Metadata ties the artifacts together, which is what lets you answer "which data trained the model that made this prediction" months later. For a broader registry discussion see MLflow.

Promotion across projects. Mature setups train in a development or staging project and serve from a production project. The flow is: train and evaluate in one project, copy or reference the model artifact into the production project's registry through a pipeline that holds permission in both, run an automated evaluation gate, then deploy. The gate matters more than the mechanism: a model that does not clear a metric threshold on a held-out set, or that regresses against the current champion, should not reach the alias.

Policy as code. Terraform provisions projects, service accounts, perimeters, endpoints and networking. Models and training runs are usually not in Terraform, because they are data and run output rather than infrastructure, so their governance lives in the pipeline and the registry. Organization policies can add guardrails at the folder level, such as restricting where resources can be created, which is one way to enforce a data-residency decision. Terraform has resources for endpoints and indexes; check the provider docs for the current names before writing a module.

Review gates in CI. Pull requests that change a pipeline, a serving container or an IAM binding should require a reviewer who owns the platform, plus automated checks: image scan, linted pipeline compile, a diff of IAM changes, and a test submission against a sandbox project. The review checklist at the end of this guide is the human half.

Region is a governance decision, not a default. Several features default to the global location or to US and European regions, and not every feature exists in every region. If you work for an employer in the Gulf or in Egypt with data-residency obligations, confirm per feature which regions are supported (for example, whether the Middle East regions you need offer the accelerator type, the endpoint type and the generative model you plan to use), and enforce the choice through organization policy rather than relying on teams remembering it. A model call routed to a different region than your training data is still a data transfer, and your auditors will treat it as one.

Observability and incident playbooks

You cannot run what you cannot see. The platform's own signals are Cloud Logging (job logs, container output, request logs when enabled), Cloud Monitoring (endpoint metrics: request count, latency, replica count, CPU and accelerator utilization, error rates), the console's endpoint dashboards, and Model Monitoring for input skew and drift (see Evidently for an open-source alternative you can run yourself, and Prometheus and Grafana if your observability lives outside Google). Build one standard dashboard and one set of alerts as a template that every new endpoint inherits, rather than letting each team invent its own.

Alert on symptoms that users feel: error rate, latency percentiles, and replica count pinned at the maximum. Alert separately on the cost signals from the previous section. Alert on the quota side too: a quota consumed above a threshold is a warning you can act on before it is a failure.

Four incident shapes cover most of what you will see. Each playbook below starts with the same two questions: what changed and what is the blast radius.

Playbook one: the endpoint is failing or slow

1. Check the endpoint's error rate and latency, and the replica count against the configured maximum. If replicas are at the maximum and latency is high, you are capacity-bound: raise the maximum if quota allows, and shed or defer load.
2. Check for a recent deployment or traffic-split change. If one exists, move traffic back to the previous deployed model first and investigate afterwards. Rolling back is a single traffic change, which is the reason you used a split in the first place.
3. If errors began with a new deployment that never became ready, read the container logs in Cloud Logging. The usual causes are a crashed server, a health route that does not answer, a wrong port (the container must listen on the port Vertex provides through AIP_HTTP_PORT), or a model file that fails to load.
4. If errors are 503 or deadline exceeded during a spike, the cause is usually scale-up lag. The fix is a higher minimum or earlier scaling, not a code change.

Playbook two: a job or pipeline will not start

1. Read the exact error. Permission denied on submission points to the caller or to actAs; a failure to pull an image points to Artifact Registry access for the service agent; a resource exhausted or quota message points to a regional quota.
2. For quota, check the quota metric and region named in the message, try an alternative region if your residency rules allow it, and file the increase.
3. For a pipeline, check that pipeline_root is set to a bucket the pipeline's service account can write, and that the bucket's location matches the job's region.

Playbook three: a credential or identity may be compromised

1. Disable the service account first; deleting it destroys the evidence and the audit trail. Rotate any secret it could read.
2. Use the Data Access audit logs to list what the identity did in the window: jobs launched, models read, buckets accessed. If a VPC Service Controls perimeter is in place, review its violation logs too.
3. Look for persistence: new service accounts, new IAM bindings, new scheduled pipelines, new endpoints, and new keys.
4. Rebuild the identity from Terraform, with fresh bindings, and review how the credential leaked.

Playbook four: the bill spiked

1. Group cost by SKU and by label for the last week. The top item is almost always an endpoint, a long-running or retried training job, or a generative workload with an unbounded loop.
2. List endpoints and their traffic; undeploy anything idle and unowned after notifying the label's owner.
3. Check for a pipeline schedule firing more often than intended, and for training jobs with retry settings that multiply failures.
4. Add the budget alert that would have caught it, and write the guard (a timeout, a trial cap, a quota) that would have prevented it.

Where Vertex AI stops

Be able to say, without defensiveness, when it is the wrong tool.

When the work is not ML-shaped. Vertex AI is not a general compute platform. A batch ETL workload belongs in Dataflow, Dataproc or BigQuery, and an arbitrary web service belongs on Cloud Run or Kubernetes. Using Vertex custom jobs as a general batch runner is possible and usually more expensive and less observable than the right tool.

When you need portability. Pipelines written in KFP are portable in principle, and the pipeline definition is reasonably open, but the Google Cloud Pipeline Components, the Model Registry, the endpoint and monitoring integrations and the IAM model are Google-specific. If you must run the same platform across clouds or on premises, the standard answer is Kubernetes with Kubeflow, KServe or BentoML for serving, and MLflow for tracking and registry. You pay for portability in operational work.

When serving needs outgrow managed endpoints. Very high request rates, unusual latency requirements, GPU sharing across many small models, or custom batching and routing logic may be cheaper and more controllable on a cluster you run, with a serving engine tuned for the workload. Managed endpoints trade control for convenience; revisit that trade when the bill or the latency ceiling says so.

When data residency cannot be met. If a feature you need is not offered in an approved region, you cannot buy your way out. The options are to wait, to choose another service, or to run an open-source equivalent in an approved location.

When your organization is on another cloud. If your data and identities live in AWS or Azure, the integration cost of Vertex AI is real: see SageMaker and Azure ML for the native equivalents, and Databricks for a cross-cloud option.

When a hosted model API is enough. Many teams reach for the full platform when they only need to call a model. The Gemini API guide covers calling Gemini directly, and for those teams a thin client may be all they require. Vertex AI's value over a plain API is governance, data controls, private networking and integration with the rest of Google Cloud, and a platform owner should be able to say whether a given team needs those things.

None of this is an argument against Vertex AI. It is the same honesty that makes your recommendations credible.

Running it as a platform for other teams

Owning Vertex AI for other teams means building a product on top of it. The product has users, an interface, a support model and a roadmap.

A paved road. Provide a golden path: a template repository per workload type (batch training pipeline, online model service, batch scoring job), each wired with the standard service-account pattern, labels, base image, CI, deployment gate, dashboard and alerts. A team should get to a working, compliant first deployment in a day. Teams that stay on the paved road need no special review; teams that leave it need a design review, which is how you keep your review load proportional.

Self-service with guardrails. Project creation, service-account creation, quota requests and endpoint provisioning should be requests that automation fulfils against policy, not tickets that a person fulfils by hand. Guardrails live in the templates, in organization policy and in CI checks, so that the compliant path is the default.

A platform SDK or wrapper. A thin internal library that sets labels, the service account, CMEK, the network and the region, and that pins the underlying SDK version, gives you one place to absorb Google's changes. When the upstream package renames a module, you change the wrapper once. The cost is that you are now maintaining a library; keep it small, and do not hide the concepts teams need to understand.

Support and documentation. Publish what the platform guarantees and what it does not: supported regions, supported machine types, the upgrade cadence, the deprecation calendar, and how to get a quota raised and how long it takes. Run office hours. Track the questions that recur and turn each into a template, a lint rule or a documentation page.

Metrics for the platform itself. Measure what makes the platform healthy: time from a new team's request to its first deployed model, share of workloads on the paved road, number of endpoints with no owner, deprecation items outstanding, percentage of pipelines running pinned and scanned images, and cost per label compared with its budget. These are the numbers you present when asked what the platform team does.

Change management. Announce upgrades and deprecations early, with dates and migration steps. Teams will forgive a breaking change they were warned about and a clear path through; they will not forgive a surprise.

The review checklist

Use these in design reviews and pull requests. A "no" is not an automatic block, but it is a question that needs an answer written down.

Identity and access

• Does each pipeline, job and endpoint run as a dedicated service account with its own narrow roles?
• Is roles/iam.serviceAccountUser granted on specific accounts, not at the project?
• Are there any downloaded service-account keys? Is CI using Workload Identity Federation?

Network and data

• Is the endpoint type appropriate: private through Private Service Connect for sensitive data?
• Is the project inside a service perimeter, and do job templates attach to the VPC?
• Are the region of the data, the job and the endpoint the same, and is it an approved region?
• Is CMEK required here, and is key ownership documented?

Supply chain

• Are images built in CI, pinned by digest, scanned, and pulled from Artifact Registry?
• Are dependencies baked in rather than installed at run time?
• Is the pipeline compiled in CI and submitted from the reviewed artifact?

Capacity and cost

• Has quota been confirmed for the accelerator, region and peak scale, and is a reservation needed?
• Are minimum and maximum replicas justified, and is scale-up time measured?
• Would batch prediction do the job? Are timeouts, trial caps and caching set?
• Does every resource carry owner, team and cost labels, and is there a budget alert?

Release and operations

• Does deployment read an alias, and is promotion gated on evaluation against the current champion?
• Is the previous version kept deployed during a soak period for rollback?
• Are logs, metrics, alerts and a dashboard defined from the template?
• Is the SDK pinned, are deprecation warnings errors in CI, and are dated shutdowns tracked?

The complete picture

Put the pieces together and the platform looks like this. A team requests a project through automation; the project arrives inside the right service perimeter, in an approved region, with dedicated service accounts, labels, budgets and an Artifact Registry repository. The team starts from a template repository. Its CI builds a scanned, digest-pinned image and compiles the pipeline. The pipeline runs as its own identity, trains on data it is permitted to read, writes a model version with lineage and evaluation results to the registry, and a gated promotion step moves an alias. Deployment reads that alias, places the model on an endpoint with sensible minimum and maximum replicas, shifts traffic in steps, and keeps the previous version warm for rollback. Monitoring, logs and cost reports exist from the first day because the template made them. Meanwhile you track the deprecation calendar, upgrade the pinned SDK monthly, and spend your own time on the work the automation cannot do: capacity planning, design reviews and the incidents that no template foresaw.

Every part of that is a decision, not a feature you switch on. The platform gives you the mechanisms (identities, perimeters, quotas, registries, traffic splits); the senior work is deciding which of them are mandatory, making the mandatory ones the default, and being able to explain each trade-off to people who would rather not think about it.

Where the series leaves you

You started this series able to run a training job and deploy a model. You can now reason about where a Vertex AI platform fails: at permissions, quotas, capacity, images, drift and dates. You can design its identity, network and tenancy model, price it, attribute its cost, and keep it current while the SDK and the product name change under you. You can also say when to leave it.

From here, the best next steps are the neighbours of this guide. Terraform is how you make the platform reproducible; Kubernetes and KServe are the portable alternative; MLflow and Feast cover tracking and features outside any single cloud; Evidently covers drift detection you control. Read the release notes and deprecations pages on a schedule, because that is the one habit that separates a platform that ages well from one that surprises its users.

Sources

• Vertex AI release notes
• Gemini Enterprise Agent Platform release notes
• Vertex AI deprecations
• Introduction to Vertex AI
• Get predictions: overview
• Vertex AI Pipelines introduction
• Access control with IAM
• Use the Vertex AI SDK for Python
• SDK 2.0 migration guide
• Agent Platform Python reference
• python-aiplatform on GitHub
• google-cloud-aiplatform on PyPI

How senior interviews on Vertex AI work

At this level nobody asks you what a Vertex AI endpoint is. Interviewers want to know whether you can design a platform on it, keep it running, and defend your choices to people who control budgets. The questions come in three families: system-design prompts, incident-response questions, and trade-off arguments. In all three, the strongest signal is that you name constraints before you name features.

One note before the questions. Google has been folding the product name into "Gemini Enterprise Agent Platform", and the Python SDK had a breaking major release (2.x). A candidate who can say, calmly, "same platform, two names, and here is what actually changed for code" has already shown more maturity than most. If the interviewer uses either name, use theirs, and mention the other once.

The thirty-second answer

Vertex AI is Google Cloud's managed platform for the machine-learning lifecycle: custom and AutoML training, a model registry, online endpoints and batch prediction, pipelines, monitoring, and hosted generative models. As a platform owner I treat it as a control plane in front of rented compute, where the real engineering is identity, quota, network boundaries, cost attribution and keeping up with a fast-moving SDK.

System-design prompts

Design an ML platform on Vertex AI for twenty teams

What a strong answer covers. Start by asking about constraints: data sensitivity, regions, the number of models in production, who owns on-call, and whether the organization already standardizes on Terraform and a CI system. Then lay out the decisions in order of how hard they are to reverse.

Tenancy first: project per team per environment, because quota, billing, IAM and perimeter membership all attach to the project. Projects are created by a Terraform module or a self-service workflow, so teams do not crowd into a shared project to avoid a ticket. Identity second: a dedicated service account per pipeline with narrow roles and no keys, CI authenticating through Workload Identity Federation, and iam.serviceAccountUser granted per account rather than per project. Network third: Private Service Connect endpoints for sensitive models, VPC Service Controls perimeters rolled out in dry-run first, and job templates attached to the VPC.

Then the golden path: template repositories for batch training, online serving and batch scoring, each with a digest-pinned scanned image, a compiled pipeline, a registry alias promotion step, a standard dashboard and a cost label set. Finally the operating model: a deprecation calendar, a monthly SDK upgrade pull request, and platform metrics such as time to first deployment.

What a weak answer does. Starts with "I would use Vertex Pipelines and Feature Store" and never mentions identity, quota or who pays.

Follow-ups to expect. "What if a team insists on a shared project?" (Offer the weaker model honestly: per-team service accounts, labels and naming, and explain that quotas are still shared.) "How do you stop a team using a personal credential?" (Policy plus detection through audit logs, and make the compliant path easier.)

A fraud model must score in under 100 ms and the data cannot leave the country

What a strong answer covers. Region comes before architecture. Confirm which Vertex AI features, accelerator types and endpoint types exist in the approved region, because not every feature is in every region and the answer changes the design. If something is missing, say what you would do: wait, pick a different component, or run an open-source server on Kubernetes in an approved location.

On the serving side: a private endpoint through Private Service Connect, a minimum of two replicas for availability, a model that loads quickly, and a measured scale-up time so you know how much headroom a spike requires. Features that must be fresh come from an online store; do not call a slow data source on the request path. For the latency budget, load test with realistic payloads, look at percentiles rather than averages, and check the payload size against the online prediction limit. Add a canary through the traffic split, and keep the previous deployment warm for rollback.

What a weak answer does. Promises "under 100 ms" without measuring, ignores cold start, and puts the model in the default region.

Design the promotion path from experiment to production

What a strong answer covers. The registry is the release contract. A training pipeline produces a model version with lineage and an evaluation report, tagged with the commit, the pipeline run and the dataset version. An automated gate compares the candidate with the current champion on a held-out set and blocks regression. Promotion moves an alias, and only the promotion pipeline's identity may do it. Deployment reads the alias. If training and serving live in different projects, a pipeline with rights in both moves the artifact, and humans do not.

Mention rollback explicitly: the previous version remains deployed for a soak period, and a traffic-split change reverts it. Mention what you would not automate, such as the first promotion of a model class that has regulatory implications.

How would you serve fifty small models cheaply?

What a strong answer covers. Do not default to fifty always-on endpoints, because each one bills for its replicas. Ask about traffic: models that are called rarely or in bulk belong in batch prediction. For the real-time ones, consider whether several share an endpoint (accepting a shared blast radius for models from one team) or whether a different serving approach, such as one multi-model server on a cluster, is cheaper. State the trade-off: consolidation reduces cost and increases coupling. Add per-model labels so you can still attribute cost.

Incident-response questions

An endpoint's latency doubled an hour ago. Walk me through it.

What a strong answer covers. Two questions first: what changed, and what is the blast radius. Check for a deployment or traffic-split change in the last hour, and if there is one, shift traffic back first and investigate later. Then check the replica count against the maximum (capacity-bound), CPU or accelerator utilization, request volume against the usual pattern, and error rate. If the replica count is at the maximum, raise it if quota allows and consider shedding load. If a new model version is slower, the rollback is the fix and the investigation is a postmortem item.

Pitfall. Digging into the model code before checking what changed. Rollback is cheap; diagnosis is expensive.

A new deployment fails with the model server never becoming ready

What a strong answer covers. The returned error is thin; the evidence is in Cloud Logging. The usual causes, in the order I would check: the container crashed on startup, the server is not listening on the port Vertex provides (the AIP_HTTP_PORT variable), the health route does not return success, the model artifact path or format is wrong, or the machine is too small to load the model. Reproduce locally by running the same image with those environment variables set. A strong candidate also says what they would change so it does not recur: a serving-container contract and a template with a smoke test that runs before deployment.

Training jobs are failing with quota errors on the day of a launch

What a strong answer covers. Identify the exact quota metric and region in the message. Check whether another region is permitted by the residency rules and has capacity. File the increase with a justification, and say plainly that approvals take time and so cannot be the plan. For the postmortem: quota belongs in capacity planning and launch checklists, with an alert when consumption passes a threshold, and where demand is predictable, a reservation. If twelve teams share a project, point out that the shared quota is the real root cause, and propose project-per-team.

You suspect a service account credential has leaked

What a strong answer covers. Disable the account, do not delete it, because the audit trail matters. Rotate secrets it could read. Use Data Access audit logs to reconstruct what the identity did: jobs launched, models and datasets read, buckets touched. Check for persistence, which means new service accounts, IAM bindings, schedules, endpoints and keys. If a service perimeter is configured, look at its violation logs, since an attempt from outside the perimeter is both evidence and a sign the control worked. Rebuild the identity from Terraform. Then ask how a key existed at all, since the standard is no keys and federated CI.

The monthly bill is double the forecast

What a strong answer covers. Group cost by SKU and label. The usual suspects, in order of how often they turn out to be responsible: forgotten endpoints, minimum replica counts, retrying or runaway training jobs, a schedule firing more often than intended, and a generative workload in a loop. Notify the label's owner before undeploying, set the budget alert that would have caught it, and add the guard that would have prevented it, such as a timeout, a trial cap or a concurrency limit. Unlabelled spend is itself a finding.

After an SDK upgrade, production code fails with import errors

What a strong answer covers. First stabilize: roll back to the pinned previous version, which is why versions are pinned. Then diagnose: the 2.x major release migrated generative modules to the Google Gen AI SDK, moved the agent surface to a separate package, and deprecated vertexai.Client. Imports of vertexai.generative_models and the other deprecated generative modules were scheduled for removal in June 2026, so they are likely the cause. Fix by moving to google-genai and the new agent package. Then prevent: pin exact versions, treat deprecation warnings as errors in CI, and run a compatibility test per major use case before bumping.

Trade-off questions: arguing it to leadership

"Why not just call a hosted model API instead of running a platform?"

Answer on value, not on technology. A thin API client is enough for many use cases, and I would say so. The platform earns its cost when you need private networking, perimeters, customer-managed keys, audit logging, a governed registry, or integration with data that already lives in Google Cloud. The decision is per workload: teams that need none of that should not carry the platform's overhead. See the Gemini API guide for the lighter path.

"Why project-per-team? That is a lot of projects."

Because the things teams argue about, which are quota, billing and permissions, attach to the project. A shared project means a shared quota and a shared blast radius, so one team's experiment can starve another's production. The cost of many projects is real, and it is paid once in automation, which is a Terraform module. Present it as buying isolation with a one-time engineering cost, not as bureaucracy.

"Should we adopt the managed service or run Kubeflow and KServe ourselves?"

Frame it as control versus operating burden. Managed means faster to a first model, built-in integration with IAM, logging and the rest of the cloud, and less to operate; it costs flexibility, a degree of lock-in and exposure to Google's change schedule. Self-hosted on Kubernetes with Kubeflow and KServe gives portability and fine control and costs a platform team's time. A good recommendation names the deciding factor for this company: multi-cloud requirements, the size of the team, the latency or cost profile of the serving workload.

"Why is the platform team asking for a monthly upgrade window?"

Because the SDK releases roughly weekly and Google publishes dated deprecations, so doing nothing is not a stable option; it just defers the work into an emergency. A predictable monthly cadence costs a small, scheduled amount of effort and removes unplanned breakage. Offer the evidence: the 2.x major release, the removal of the older generative modules, and the Feature Store and Explainable AI shutdown dates in 2027.

"Can we cut the endpoint cost without hurting users?"

Offer a ranked list, not a single trick. Remove idle endpoints. Move non-interactive workloads to batch prediction. Right-size machine types from utilization data. Reduce the minimum replica count only for models where an availability promise does not apply. Use committed-use discounts or reservations for steady GPU demand once you have utilization history. Say what each option risks, and who must agree.

Questions about Vertex AI specifics

What is the difference between a service agent and a custom service account? A service agent is a Google-managed identity that the platform uses to do things on your behalf, such as pulling an image. The custom service account is the identity your own code runs as, and it is the one that reaches your data. Least privilege applies to both, and the custom account is where you have the most control.

What does iam.serviceAccountUser actually grant? The ability to act as that service account, which means running any code with its permissions. Treat it like a grant of the account's own power, and give it on individual accounts only.

Private Service Connect or a public endpoint? Public dedicated endpoints suit models that serve internet-facing workloads and have a path you want to expose. Private Service Connect suits sensitive workloads that should have no route from the internet. Both are recommended over the older shared public endpoint, and any design on older VPC peering should be checked against the current docs first.

Where do you put the production alias in the registry, and who may move it? On the version that is serving, and only the promotion pipeline's identity may move it. Deployment reads the alias.

What is a VPC Service Controls perimeter for, in one sentence? It stops data leaving the protected projects and APIs even when an identity is compromised, because calls from outside the perimeter are refused.

When would you choose batch prediction? When the result is not needed in real time. It uses machines only for the duration of the job and has no idle endpoint, so it is usually the cheapest option for scheduled scoring.

What changed in the SDK 2.x release? Generative modules moved to google-genai, the agent surface moved to a separate package (google-cloud-agentplatform, imported as agentplatform), vertexai.Client is deprecated in favour of agentplatform.Client, agent_engines was renamed runtimes, and agent frameworks no longer read project and location from aiplatform.init().

Behavioural questions you should prepare

Senior interviews include stories. Prepare three, each with the situation, your decision, the trade-off you made and what you would change.

• A time you said no. Perhaps a team wanted a shared project or a long-lived key. The strong version describes the harm you saw, the alternative you offered, and how you made the compliant path easier.
• A time you absorbed a breaking change. The SDK major release is a ready example. Describe how you found out, how you bounded the blast radius, and what you changed in your process.
• A time you reduced cost without hurting users. Give numbers: what was running, what you measured, what you changed, and the saving.

What a strong answer covers that a weak one misses

• Constraints before components. Strong candidates ask about residency, scale, ownership and budget. Weak candidates start naming services.
• Identity. Strong candidates raise service accounts, actAs and keys without being prompted. Weak candidates leave it until asked.
• Quota and capacity. Strong candidates treat them as inputs to design. Weak candidates discover them as errors.
• Rollback. Strong candidates say how they would undo a change before describing the change.
• Dates. Strong candidates know the platform has a deprecation calendar and say who tracks it.
• Honest limits. Strong candidates say when Vertex AI is the wrong tool, and what they would use instead.
• Verification. Strong candidates say "I would check the current docs for that" when they are unsure, rather than inventing a flag or a limit.

Questions to ask the interviewer

Good questions signal that you think like an owner.

• How many projects, and how many teams, share the platform today, and where does quota bite?
• Who owns on-call for endpoints, and who owns the cost of an idle one?
• Is the SDK pinned across services, and how are deprecations tracked?
• Which regions are approved, and has anyone verified that the features you need exist there?
• What does a team have to do to get a first model into production, and how long does it take?

Sixty-second self-test

Answer aloud, without notes.

1. Why is project-per-team the default tenancy model?
2. What is the difference between a service agent and the custom service account, and which one is your code?
3. Name three reasons a deployment might never become ready.
4. A job fails at launch with a quota error. What are your first three actions?
5. How do you roll back a bad model deployment, and what must still be true for that to work?
6. Why does a nightly scoring workload belong in batch prediction?
7. Name three changes in the 2.x SDK, and what you would do in CI to catch the next one.
8. When is Vertex AI the wrong choice?
9. What do you check first when the bill doubles?
10. How do you disable a leaked service account, and why not delete it?

How to use this file

This file is for the person who owns Vertex AI as a platform. It lists the traps that appear only once many teams and many models are involved, the habits that keep a platform manageable, and the checklists you can paste into pull request templates and design documents. The traps are written as symptom, cause and fix, because that is the order in which you meet them.

Throughout, two reminders. First, the product is being renamed (Gemini Enterprise Agent Platform), the Python SDK had a breaking 2.x release, and release cadence is roughly weekly, so verify any flag or API name against the current documentation before putting it into a standard. Second, when this file says "check the docs", it means that literally: several limits and role names are the sort of thing that changes without a headline.

Production traps that appear at scale

The quota nobody knew was shared

Symptom. One team's training job fails or queues, and the team swears nothing changed on their side.

Cause. Quotas belong to the project and are set per region and accelerator type. Another team in the same project launched a sweep and consumed it.

Fix. Move teams to project-per-team so quota is per team. In the meantime, alert when consumption of a key quota passes a threshold, and put concurrency limits in the shared pipeline templates so no single run can take everything. File quota increases during planning, because approvals take human time.

The endpoint that billed all month for nothing

Symptom. A line item for node-hours on an endpoint with almost no requests.

Cause. Endpoints bill for replicas whether or not they are called. Someone deployed for an experiment and moved on.

Fix. A scheduled audit that lists endpoints across all projects with their owner label, replica count and recent request count. Notify the owner, then undeploy after a grace period. Make owner a required label in templates so unowned endpoints are detectable. Teardown is endpoint.undeploy_all() followed by endpoint.delete().

Minimum replicas of one

Symptom. A short outage during a routine update or a replica failure, with errors rather than slow responses.

Cause. With a minimum of one, a single replica failing, or being replaced during a rollout, leaves zero capacity.

Fix. Anything with an availability promise runs at least two replicas. The doubled baseline cost is a conscious decision with a named owner, not a default.

Autoscaling that is too slow for the traffic

Symptom. Errors or high latency for a minute or two when traffic ramps, which then resolve themselves.

Cause. New replicas must be provisioned, pull the image, load the model and pass the health check. If the model takes a long time to load, scaling is slower than the ramp.

Fix. Measure the time from "replica requested" to "replica serving". Keep headroom through a higher minimum, or move to a smaller, faster-loading model artifact. Do not tune the utilization target before you know what the load time is.

The rollback that is not there

Symptom. A bad release, and nothing to roll back to.

Cause. The promotion process undeployed the previous version as soon as the new one passed a health check.

Fix. Keep the previous deployed model for an agreed soak period, move traffic by percentage, and rehearse the rollback. A traffic split is a rollback only if the old version is still there and healthy.

Everyone can act as the powerful account

Symptom. An audit finds that thirty people can run code as a service account with broad data access.

Cause. roles/iam.serviceAccountUser was granted at project level, or to a broad group, because it was the quickest way to unblock someone.

Fix. Grant it on individual service accounts, to specific principals. Review these grants like grants of the underlying power, because that is what they are. Use dedicated, narrow service accounts per pipeline so that "can act as this account" means very little.

Keys that were supposed to be temporary

Symptom. A downloaded service-account key turns up in a repository, a laptop or a CI variable.

Cause. Someone needed to authenticate from outside Google Cloud and took the shortest route.

Fix. Use Workload Identity Federation for CI and for workloads running elsewhere, and run an organization policy that blocks key creation where your policy allows it. Scan repositories for key files, and treat any found key as compromised.

Perimeter turned on, everything breaks

Symptom. After enabling VPC Service Controls, pipelines, notebooks or vendor tools fail with permission-style errors that look like IAM problems but are not.

Cause. Legitimate flows now cross the perimeter, and nobody wrote ingress or egress rules for them.

Fix. Enable the perimeter in dry-run first, read the logged violations, write the rules for the legitimate ones, and then enforce. When a failure looks like IAM but the IAM is right, check the perimeter logs.

Packages installed at every run

Symptom. A pipeline that worked yesterday fails today, or behaves differently, with no change to your code.

Cause. Components with packages_to_install run pip install at start, so a new upstream release changed what you got.

Fix. Bake dependencies into an image you build in CI, pin versions, scan the image, and use it as the component's base image. Pin by digest in production specs.

Region drift

Symptom. Slow jobs, odd errors about bucket locations, or an auditor asking why data crossed a border.

Cause. A bucket in one region, a job in another, and a generative call defaulting to a third. Several features default to the global location or to US and European regions.

Fix. Make region an explicit parameter in every template, enforce allowed locations with organization policy where it applies, and verify per feature that the approved region supports it. Do this before a launch, not after.

Model names hard-coded in code

Symptom. Generative features suddenly return not-found errors for a model that worked last quarter.

Cause. Model versions retire on published dates. Gemini 1.5 access ended on 24 September 2025, for instance, and several partner model versions have had shutdown dates in 2026.

Fix. Keep model identifiers in configuration, read the model versions page when you set them, and put retirement dates on the deprecation calendar.

Deprecated imports in code nobody touches

Symptom. After an upgrade, an import error for vertexai.generative_models or a similar module, or an attribute error on vertexai.Client.

Cause. Those modules were deprecated in 2025 and scheduled for removal in June 2026, and the 2.x SDK release changed the client and agent surface.

Fix. Migrate to google-genai and agentplatform. Pin the SDK, and run CI with deprecation warnings turned into errors so the next one is found in a pull request.

Features with shutdown dates

Symptom. A team builds new work on the legacy Feature Store or on Vertex Explainable AI.

Cause. Nobody told them. Both have published shutdown dates in 2027 (legacy Feature Store on 17 February 2027, Explainable AI on 16 March 2027).

Fix. Publish the deprecation calendar, and put a lint or review rule in templates that blocks new use of deprecated services. See Feast if you want a feature store independent of the cloud vendor.

Alerts on causes rather than symptoms

Symptom. Noisy pages for CPU spikes while real user errors go unnoticed.

Cause. Alerts built from whatever metric was easy to chart.

Fix. Alert on what users feel: error rate, latency percentiles, replica count stuck at the maximum. Keep cost and quota alerts separate so they can be routed to the people who can act on them.

Platform-ownership habits

Write standards in API resource names, not console labels. The product name has already changed once. Resource paths, API names and IAM concepts are more stable than the console, so document those.

Pin everything and upgrade on a schedule. Exact SDK versions in lock files, digests for images, and a monthly pull request that bumps versions, runs the compatibility tests and reads the release notes. Weekly releases make "latest" an unreviewed change every week.

Keep a deprecation calendar with owners. The deprecations page is a list of dated promises. Copy the dates that affect you into a tracker, assign an owner, and set the reminder three months ahead of each.

Make the compliant path the easy path. A template repository that gives a team a labelled, scoped, scanned, monitored deployment in a day beats a policy document. Teams on the paved road need no special review; spend your attention on those who leave it.

Own the quota conversation early. Maintain a living spreadsheet or dashboard of quota by project, region and accelerator type, with the next planned peak. Raise requests when you plan, not when you fail.

Label everything, then report on the labels. team, service, env and owner on every job, model and endpoint. Export billing to BigQuery, group by label, and report the unlabelled share as a metric you drive to zero.

Use batch prediction by default for non-interactive work. Ask "does this need to be real-time?" for every request. A nightly job on an always-on endpoint is the most avoidable cost on the platform.

Rehearse failure. Once a quarter, run a rollback drill, disable a test service account to see what depends on it, and practise reading the audit logs. Incident time is a bad moment to learn where the logs are.

Verify before standardizing. Role names, limits, region availability and flag names shift, especially mid-rename. Before an internal standard says "grant role X" or "payload limit Y", open the current documentation and cite the page.

Say no with an alternative. Refusing a long-lived key is only useful if you also offer federated authentication that is easy to set up. Refusals without alternatives become workarounds.

Keep a short list of what the platform does not do. Teams trust a platform that says plainly which regions, machine types and features it supports. See Terraform for making that list executable.

Review checklist for pull requests

Paste into a pull request template for any change touching pipelines, serving containers, Terraform for ML projects, or IAM.

Identity

• The job, pipeline or endpoint runs as a dedicated service account, not a default or a person.
• New IAM bindings are the narrowest role on the narrowest resource, with a justification in the description.
• No service-account keys, tokens or passwords in code, arguments, YAML or images.
• Any new iam.serviceAccountUser grant is on a specific account.

Supply chain

• Image is built in CI, pulled from Artifact Registry, pinned by digest and scanned (see Trivy).
• No packages_to_install or runtime pip install for production components.
• No latest tags in production specs.
• The pipeline was compiled in CI and the submitted artifact is the reviewed one.

Capacity and cost

• Quota was checked for the accelerator and region at peak.
• Min and max replicas are justified, with scale-up time measured.
• Timeouts, trial caps and a concurrency limit are set on training work.
• Labels team, service, env and owner are present.
• Batch prediction was considered if the workload is not interactive.

Release and operations

• Deployment reads an alias, and the promotion gate compares with the current champion.
• The previous version stays deployed for a soak period.
• Dashboard and alerts come from the template and cover errors, latency and replicas.
• SDK version is pinned and deprecation warnings fail CI.
• No imports from vertexai.generative_models, vertexai.language_models, vertexai.vision_models, vertexai.tuning or vertexai.caching.

Review checklist for design documents

Ask these of any design document that proposes a new model service or pipeline on the platform.

Problem and shape

• Does the workload need real-time inference, or would batch prediction meet the need?
• What are the latency, availability and throughput targets, and how will they be measured?
• What data does it read, how sensitive is it, and where must it stay?

Boundaries

• Which project, which region, and which service perimeter? Has the region been checked for the features the design needs?
• Is the endpoint public, dedicated public, or private through Private Service Connect, and why?
• Is CMEK required, and who owns the key?

Operations

• Who is on call, and what is the rollback procedure?
• What are the dashboards and alerts, and which of them page someone?
• How is the model retrained, how is drift detected, and who decides to retrain?

Evolution

• Which parts of the design rely on features with published shutdown dates?
• How are SDK and image upgrades handled?
• What would make you leave Vertex AI for this workload, and what is the exit cost?

Cost

• What is the expected monthly cost at steady state and at peak, and who approves it?
• What is the plan to detect and remove idle resources?

Practice cards

Each is a ten-minute exercise you can run in a sandbox project.

1. Quota map. List quotas for the accelerator types you use in each approved region. Mark the three tightest against last month's peak.
2. Idle endpoint sweep. Write a script that lists endpoints across projects and flags those with no owner label. Run it, and file a ticket for each hit.
3. actAs audit. Find every principal with roles/iam.serviceAccountUser at project level. Replace each with a per-account grant or remove it.
4. Rollback drill. Deploy two versions to one endpoint, send test traffic, shift the split to ten percent and then back. Record the elapsed time.
5. Dry-run perimeter. Put a sandbox project in a VPC Service Controls perimeter in dry-run mode, run a pipeline, and read the violations.
6. Deprecation grep. Search your repositories for removed or deprecated imports and for gcr.io. Count hits by team.
7. Cost by label. Export billing to BigQuery and chart last month's Vertex AI spend by team. Note the unlabelled share.

Operational recipes

List endpoints and what is deployed to them:

gcloud ai endpoints list --region=us-central1
gcloud ai endpoints describe ENDPOINT_ID --region=us-central1

Deploy a model to an existing endpoint with explicit capacity:

gcloud ai endpoints deploy-model ENDPOINT_ID --region=us-central1 \
  --model=MODEL_ID --display-name=DEPLOYMENT_NAME \
  --machine-type=n1-standard-4 --min-replica-count=2 --max-replica-count=6

Canary a new version at ten percent of traffic using the SDK:

from google.cloud import aiplatform

aiplatform.init(project="ml-prod", location="us-central1")
endpoint = aiplatform.Endpoint("projects/NUM/locations/us-central1/endpoints/ENDPOINT_ID")
model = aiplatform.Model("projects/NUM/locations/us-central1/models/MODEL_ID@8")

model.deploy(
    endpoint=endpoint,
    machine_type="n1-standard-4",
    min_replica_count=2,
    max_replica_count=6,
    traffic_percentage=10,
)

Tear down an experiment endpoint so it stops billing:

endpoint.undeploy_all()
endpoint.delete()

Submit a pipeline as a dedicated identity, with caching on:

from google.cloud import aiplatform

aiplatform.PipelineJob(
    display_name="churn-train",
    template_path="pipeline.yaml",
    pipeline_root="gs://ml-churn-prod-pipeline-root",
    enable_caching=True,
    labels={"team": "churn", "env": "prod", "owner": "ayesha"},
).submit(service_account="churn-train@ml-churn-prod.iam.gserviceaccount.com")

Make deprecation warnings fail a test run:

python -W error::DeprecationWarning -m pytest tests/

Pin the SDK in a requirements file rather than installing the latest release:

google-cloud-aiplatform==2.3.0
google-genai
kfp

In practice, pin google-genai and kfp to exact versions too, using the versions your compatibility tests passed against.

When something looks wrong, in order

1. What changed? A deployment, a traffic-split change, an SDK bump, an IAM change, a quota change. Check the audit log and the release history before anything else.
2. What is the blast radius? One endpoint, one team, one project, or everyone. This tells you who to call.
3. Can it be undone cheaply? A traffic move, a pinned version, a disabled account. Undo first, diagnose after.
4. Is it permissions, quota, capacity, container or code? The error text usually names which one. Read it fully.
5. What would have caught this earlier? Write the alert, the template change or the guard, and add it to the platform backlog.

A closing checklist for the platform owner

• Every team has its own project, inside the right perimeter, in an approved region.
• Every workload runs as a dedicated service account, with no keys anywhere.
• Images are built in CI, pinned, scanned and pulled from Artifact Registry.
• Quota is mapped, planned and alerted on.
• Every resource carries owner and cost labels, and idle endpoints are swept weekly.
• Promotion goes through an alias and a gate, and rollback has been rehearsed.
• The SDK is pinned, upgraded monthly and tested in CI with warnings as errors.
• The deprecation calendar has owners and dates.
• You can say, per workload, why it is on Vertex AI and what would make it leave.