The Complete Azure OpenAI Guide

This is part one of three. It covers everything you need to do real work with Azure OpenAI as a first-time user, not a teaser. By the end you can create the Azure resources that host a model, deploy a model under a name of your choosing, call it from curl and from Python, sign in without a password in your code, stream an answer, hold a conversation, turn text into embeddings, and read the errors the service sends back. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task where it makes sense. Do them as you go. They take a few minutes each, and the service only starts to feel predictable once you have watched your own deployment answer, fail, and answer again.

One honest note before we start. Azure OpenAI is a managed cloud service, not software you install, so "the version" does not mean what it does for a command-line tool. This guide teaches the v1 API (generally available since August 2025), the openai Python package 3.x, and Azure CLI 2.90. Where Microsoft's own pages still show an older pattern, we say so and show the current form.

What Azure OpenAI is, and why people choose it

Azure OpenAI is Microsoft's hosting of OpenAI's models (the GPT family, the embedding models, image and audio models) inside your own Azure subscription. You send text to an HTTPS endpoint that belongs to you, a model reads it, and you get text back. Microsoft's current name for it is "Azure OpenAI in Microsoft Foundry Models". People still say "Azure OpenAI", and so does this guide.

If you have called the OpenAI API directly, the experience is almost the same. The difference is everything around the model. Your requests go to an endpoint that sits in your Azure subscription, you pay on your Azure bill, you control who may call it with the same identity system you use for the rest of Azure, and Microsoft's data commitments apply. Those commitments, in Microsoft's own words, are that your prompts, completions, embeddings and training data are not available to other customers or to OpenAI, and are not used to train the foundation models. For an employer in the Gulf or Egypt that has a security team, a procurement process and a rule about where data may live, that wrapper is often the reason Azure OpenAI is chosen over the public API.

The diagram is the whole idea, and the rest of this guide fills in each box. What you can do once the path works:

You need little to follow along: an Azure subscription you are allowed to create resources in, a terminal, and Python 3.10 or newer. If your employer manages your subscription, ask for permission to create a resource group and an Azure OpenAI resource in it; the rest of the guide depends on that.

The problem it solves, and what came before

Large language models are expensive to run. A serious model needs specialised hardware, careful capacity planning, and a team that keeps it patched and fast. For most companies, running one themselves is out of the question, so the first way to use a frontier model was a public web API: create an account, get a key, send requests.

That worked for experiments and became awkward for companies. A key pasted into a script has full power and no owner. Data leaves for a service your security team has not reviewed. There is no way to say "only this application may call the model" in the language the organisation already uses for permissions. Billing sits on a separate card. And the questions auditors ask, such as where the data is processed and who can see it, have answers that live outside your cloud contract.

Azure OpenAI answers those questions by putting the same family of models behind the controls Azure already has. The model is a resource like a database or a storage account. You create it in a region, give it a name, restrict its network if you need to, and grant roles to people and applications. Microsoft's billing, monitoring and policy tooling all see it.

Two consequences shape everything that follows, so notice them now.

You do not call a model, you call your deployment of a model. In the public API you write model="gpt-5.1". In Azure OpenAI you first create a deployment of that model, give the deployment a name you choose, and write that name in the model field. Mix the two up and you get a 404. This single difference causes more first-day failures than anything else, and we will return to it more than once.

Capacity is something you are assigned, not something you assume. Every deployment has a rate limit measured in tokens per minute, drawn from a quota that your subscription holds per model and region. Hit it and the service answers with HTTP 429. Nothing is broken when that happens; you asked for more than you were allotted. We will look at the numbers when we get there.

The mental model: five nouns

Azure OpenAI has more settings than you will ever need on day one, but it rests on a few nouns. Learn these five and every screen and error message makes sense.

The resource. An Azure OpenAI resource is an object in your subscription (technically of type Microsoft.CognitiveServices/accounts) that lives in one region. It owns an endpoint, two access keys, network settings and an optional managed identity. There are two flavours. The classic one has kind: OpenAI. The newer one, a Foundry resource, has kind: AIServices; it is a superset that adds other providers' models, agents and projects. Upgrading from the first to the second keeps your name, endpoint, keys and settings, and Microsoft is gradually upgrading eligible resources on its own. For a beginner the difference is small, and the commands in this guide work against both. The one thing to remember is that a resource has a custom subdomain, a name that becomes part of its URL. Passwordless sign-in requires one.

The endpoint. Your endpoint is a URL of the form https://YOUR-RESOURCE-NAME.openai.azure.com. Every call goes to a path under it. The current API lives under /openai/v1/, so the base URL you will use throughout is https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/.

The model and its version. A model is an entry in Microsoft's catalogue, such as gpt-5.1. A model version is a dated release of it, such as 2025-11-13. Models also come in tiers: a family like GPT-5 has big and small variants, with names like mini and nano for the smaller, cheaper ones.

The deployment. A deployment is a named instance of one model version on one resource, with a deployment type and a capacity. The name is yours: my-gpt, chat-prod, support-bot. That name is what API calls address. A resource can hold up to 32 standard deployments, and you can run the same model under several deployment names with different capacities or guardrails.

The token. Models do not read characters or words; they read tokens, which are fragments of text, often a short word or part of a longer one. You are billed per token, limits are expressed in tokens, and a request has input tokens (what you send) and output tokens (what the model writes). For ordinary English, a token is roughly three quarters of a word, so a hundred words is a bit over a hundred tokens. Arabic and other non-Latin scripts often use more tokens per word, which matters when you estimate cost for Arabic content.

Two planes also matter, and you will see both in the docs. The control plane is Azure Resource Manager at management.azure.com: creating resources, deployments, quota and guardrail policies. The data plane is the model API at your endpoint: sending prompts, uploading files, running batches. The Azure CLI commands in this guide use the control plane. Your application code uses the data plane. Permissions are separate for each, which is why someone can be an Owner of the resource and still be refused when calling the model. We will see that exact trap in the errors section.

Deployment types, quota and where your data is processed

When you create a deployment you choose a deployment type. The names look intimidating, but they answer two simple questions: where may the request be processed, and how is capacity billed?

On the first question there are three scopes. Global deployments let Microsoft process the request in any Azure region, which gives the newest models, the highest default quota and the lowest price. Data Zone deployments keep processing inside a zone: the United States, the European Union, or Asia-Pacific. Standard (regional) deployments process within the geography of your resource. In every case, your stored data at rest, such as uploaded files, stays in the geography of the resource. That last sentence is worth reading twice if your employer has a residency rule, because "processing location" and "storage location" are different questions with different answers.

On the second question, most deployments are pay-as-you-go: you pay per token and draw on a quota. The alternative is provisioned capacity, where you reserve throughput units and pay for them by the hour whether you use them or not. Provisioned capacity suits steady, high-volume production workloads and is a mid-level topic. A third type, batch, runs large asynchronous jobs at a discount. As a beginner you want one thing: Global Standard. In code or in the CLI it is the SKU name GlobalStandard. The other names you will see in the portal are DataZoneStandard, Standard, GlobalBatch, DataZoneBatch and the provisioned ones, and you can leave them alone for now.

Now the numbers. Quota is an allowance of tokens per minute (TPM) that your subscription holds for a given model and deployment type. When you create a deployment you assign part of that quota to it, and that slice becomes the deployment's rate limit. For standard deployments, one unit of capacity equals 1,000 TPM, so a capacity of 10 means 10,000 tokens per minute. A requests-per-minute limit is derived from it: older chat models get six requests per minute for each 1,000 TPM, and most newer models get one request per minute for each 1,000 TPM. The service also looks at short windows, so a deployment limited to 600 requests per minute can be throttled if it receives more than ten in a single second.

Microsoft has been changing how quota works. Quota now comes in tiers: a free tier (Tier 0) and Tiers 1 to 6, which move up automatically based on usage and payment history. Starting after 7 May 2026, quota has also been moving to a subscription-level pool, where Global Standard shares one pool per model and version across all regions. Exact numbers differ by model and change over time, so do not memorise them. Look at the Quota page in the portal for your own subscription, and treat any figure in a tutorial as an example. As an example only, the free tier lists 200,000 TPM for gpt-4.1-mini and 500,000 TPM for gpt-5-mini.

The practical beginner lesson is modest. Start with a small capacity (10 units is plenty for learning), expect to see a 429 when you loop too fast, and remember that you can raise a deployment's capacity from the portal as long as unassigned quota remains in the pool. Changes can take about 15 minutes to apply.

Setting up: an account, the Azure CLI and a Python environment

Azure OpenAI is not installed, so setup means three things: an Azure subscription, the command-line tool for managing it, and a language library for calling the model. Many people do the first part in the portal. Doing it from the command line instead gives you something you can paste into a note, repeat next month, and eventually turn into infrastructure-as-code.

Install the Azure CLI, called az. On macOS use Homebrew:

brew update && brew install azure-cli

On Windows:

winget install --exact --id Microsoft.AzureCLI

On Debian or Ubuntu Linux:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Check that it works:

az version

You should see a JSON block listing azure-cli with a version number; this guide was prepared against 2.90. Then sign in:

az login

A browser window opens, you sign in with your work or personal Azure account, and the terminal prints the subscriptions you can use. If you belong to several organisations, name the one you want with az login --tenant <TENANT_ID>. Confirm where you landed:

az account show --query "{subscription:name, tenant:tenantId}" --output table

If the subscription is not the one you intended, switch with az account set --subscription "<name or id>" before creating anything. Creating a resource in the wrong subscription is an easy mistake and an annoying one to undo.

Now create a Python environment and install the two libraries you need. The openai package is the client, and azure-identity handles passwordless sign-in:

python3 -m venv .venv
source .venv/bin/activate
pip install openai azure-identity

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1. The current openai release at the time of writing is in the 3.x line. A virtual environment keeps these packages away from the rest of your system, and it is the habit to build from day one.

Creating a resource and your first deployment

You will now create the two things that make the endpoint exist: a resource and a deployment. The commands below are the ones in Microsoft's documentation, adjusted to a current model.

First a resource group, a folder-like container that makes cleanup easy because deleting it deletes everything inside:

az group create --name OAIResourceGroup --location eastus

Pick the region with care. A resource lives in one region, and the models and quota available differ from region to region. Check Microsoft's model availability pages for the region closest to you that also satisfies your residency rule. If you are in the Middle East, remember that not every model is offered in every nearby region, and that the processing location depends on the deployment type you pick, as described earlier.

Now the resource itself:

az cognitiveservices account create \
  --name MyOpenAIResource \
  --resource-group OAIResourceGroup \
  --location eastus \
  --kind OpenAI \
  --sku s0 \
  --custom-domain MyOpenAIResource \
  --yes

Resource names must be unique across Azure because the name becomes part of the URL, so choose your own and use the same string for --custom-domain. Always pass --custom-domain. Without it, passwordless sign-in does not work, and Microsoft documents a known problem where resources created from the CLI without a proper subdomain misbehave with batch jobs. The --kind OpenAI flag makes a classic Azure OpenAI resource. If you want a Foundry resource, the kind is AIServices; for learning, OpenAI is simpler.

Read back the two values you will use in code:

az cognitiveservices account show \
  --name MyOpenAIResource --resource-group OAIResourceGroup \
  --query properties.endpoint --output tsv

az cognitiveservices account keys list \
  --name MyOpenAIResource --resource-group OAIResourceGroup \
  --query key1 --output tsv

The first prints the endpoint, something like https://MyOpenAIResource.openai.azure.com/. The second prints a secret key. Treat that key like a password; we will discuss storing it shortly.

Now deploy a model:

az cognitiveservices account deployment create \
  --name MyOpenAIResource \
  --resource-group OAIResourceGroup \
  --deployment-name my-gpt \
  --model-name gpt-5.1 \
  --model-version "2025-11-13" \
  --model-format OpenAI \
  --sku-name GlobalStandard \
  --sku-capacity 10

Read it as a sentence. On the resource MyOpenAIResource, create a deployment called my-gpt, which is the model gpt-5.1 at version 2025-11-13, as Global Standard, with a capacity of 10 units (10,000 tokens per minute). If it fails with a message about quota or availability, either the region lacks that model or your subscription has no quota for it. List what you have and try a different region or model:

az cognitiveservices account deployment list \
  --name MyOpenAIResource --resource-group OAIResourceGroup --output table

The deployment you just created should appear in the table with its model and capacity. Remember that a newly created deployment can answer with a "deployment does not exist" error for a few minutes; the service says so in the message.

The same steps work in the portal. Open Microsoft Foundry at ai.azure.com, select your resource, go to the model catalogue, choose the model, and click Deploy. The portal shows a default deployment name equal to the model name, and many beginners keep it. That is fine, as long as you remember that the name in the box is now the name your code must use. There are two portal experiences, the new Foundry and "Foundry (classic)", switched with a toggle, and screenshots in tutorials may match either. The CLI does not change, which is another reason to learn it.

When you finish experimenting, clean up. Deleting a resource group removes everything in it:

az group delete --name OAIResourceGroup --yes

One caution. A deleted Azure OpenAI resource is soft-deleted. If you delete a resource through the API or CLI while deployments still exist, its quota stays locked for 48 hours unless you purge it. Delete the deployments first, then the resource, or purge the deleted resource.

Your first call

You now have an endpoint, a key and a deployment. Put the key and endpoint into environment variables so they never appear in your code. On macOS or Linux:

export AZURE_OPENAI_API_KEY="paste-your-key-here"
export AZURE_OPENAI_ENDPOINT="https://MyOpenAIResource.openai.azure.com"

Start with curl, because it shows you the raw request without any library in the way:

curl -X POST "$AZURE_OPENAI_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "api-key: $AZURE_OPENAI_API_KEY" \
  -d '{"model": "my-gpt", "input": "This is a test."}'

Look at its parts. The URL is your endpoint plus /openai/v1/responses. The key travels in an api-key header. The body is JSON with two fields: model, which holds your deployment name, and input, which holds the prompt. The reply is a larger JSON document containing, among other things, an id, a status of completed, token counts under usage, and an output list whose first message holds the text.

Now the same call in Python, using the plain OpenAI client pointed at your endpoint. This is the current pattern for the v1 API:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    base_url=os.getenv("AZURE_OPENAI_ENDPOINT") + "/openai/v1/",
)

response = client.responses.create(
    model="my-gpt",  # the DEPLOYMENT name, not the model name
    input="Explain what a token is in one sentence.",
)

print(response.output_text)

Run it with python hello.py. You should see one sentence about tokens. If you do, you have completed the hard part, and everything else in this guide is variations on this call.

Notice also the API we called. responses.create is the Responses API, which is now the recommended way to talk to Azure OpenAI models. The older Chat Completions API still works, and you will see it everywhere in tutorials, so we cover it below. The shape of the answer differs slightly: the Responses API gives you response.output_text as a convenience, while Chat Completions nests the text under choices[0].message.content.

Signing in without a password: keys versus Entra ID

You have been using an API key. A key is simple, and it is also the weakest option. It grants full access to the data plane with no role checks, anyone who holds it can use your quota and your bill, and it is easy to leak into a repository or a screenshot. Azure gives you a better way: Microsoft Entra ID, Azure's identity system, where the caller proves who it is and Azure checks a role.

With Entra ID, your code holds no secret. Locally, it borrows the identity you used for az login. In Azure, it uses a managed identity attached to the service that runs your code. The library that does this is azure-identity, and its workhorse is DefaultAzureCredential, which tries several sources in order (environment variables, a managed identity, your Azure CLI login, and more) until one works.

The code changes by only a few lines:

from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)

client = OpenAI(
    base_url="https://MyOpenAIResource.openai.azure.com/openai/v1/",
    api_key=token_provider,
)

response = client.responses.create(model="my-gpt", input="Say hello.")
print(response.output_text)

The only odd line is api_key=token_provider. The api_key parameter accepts a function as well as a string. The client calls the function before each request, gets a fresh short-lived token, and sends it as an Authorization: Bearer header. You never handle tokens yourself, and they refresh automatically. The string https://ai.azure.com/.default is the scope, the audience the token is meant for. Older documentation uses https://cognitiveservices.azure.com/.default; current docs use the Foundry one shown above, and a token for one audience does not work on another, so copy the scope from current documentation.

Two things must be true for this to work, and each is a classic first-week failure.

First, your resource needs the custom subdomain you set with --custom-domain. Calls to a regional endpoint instead of your own subdomain return 401.

Second, your identity needs a role on the resource. Being the Owner of the subscription is not enough. Owner and Contributor are control-plane roles: they let you manage the resource but not call the model through Entra. The role for inference is Cognitive Services OpenAI User, and on a Foundry project the equivalent is Foundry User. Assign it to yourself with the CLI:

az role assignment create \
  --role "Cognitive Services OpenAI User" \
  --assignee "$(az ad signed-in-user show --query id --output tsv)" \
  --scope "$(az cognitiveservices account show \
      --name MyOpenAIResource --resource-group OAIResourceGroup \
      --query id --output tsv)"

Role assignments can take up to five minutes to propagate, so a 403 right after assigning one is normal. Wait a few minutes before debugging. Note that the Cognitive Services Contributor role, despite its name, cannot make Entra inference calls either; it manages the resource and its keys.

You can check that Azure will hand you a token, which separates "I am not logged in" from "I lack a role":

az account get-access-token --resource https://ai.azure.com --query expiresOn --output tsv

If that prints a date, your login works and any remaining failure is about roles or the endpoint.

So which should you use? For learning on your own laptop, a key is acceptable if you keep it in an environment variable and never commit it. For anything that a team or a company depends on, use Entra ID, and eventually turn keys off entirely. That hardening step is a senior topic, and the point for now is to build the habit: your first real application should use DefaultAzureCredential.

Everyday calls: instructions, conversations, streaming and Chat Completions

With the connection solved, most of your work is choosing what to send. This section covers the calls you will make daily.

Giving the model instructions. A model behaves according to its input. The Responses API lets you pass a separate instructions string that sets the role and rules, and an input that holds the user's request:

response = client.responses.create(
    model="my-gpt",
    instructions="You are a support assistant for a bank. Answer in two sentences, in plain language.",
    input="How do I reset my online banking password?",
)
print(response.output_text)

Instructions are the main lever you have. Be specific about audience, length, format and what to do when the model is unsure. "Answer only from the text below; if the answer is not there, say you do not know" is worth more than any amount of hoping.

Holding a conversation. Models are stateless, so a second request knows nothing about the first unless you tell it. The Responses API gives you two ways to carry context. The first is to chain by response ID:

first = client.responses.create(
    model="my-gpt",
    input="My name is Aya and I work in Cairo.",
)

second = client.responses.create(
    model="my-gpt",
    input="Where do I work?",
    previous_response_id=first.id,
)
print(second.output_text)

Here the service remembers the first exchange on your behalf. By default it stores responses for 30 days, which is how previous_response_id can work. If you must not have the service store anything, pass store=False and use the second approach, which is to send the history yourself: keep a Python list of messages, append the model's output and your new question, and send the whole list each time. You can also fetch a stored response with client.responses.retrieve(first.id) and remove it with client.responses.delete(first.id). Either way, remember that each turn re-reads the entire history, and you are billed for those input tokens every time. Long chats get more expensive per message as they grow.

Streaming. A long answer can take many seconds to finish. Streaming sends the text as it is generated, so a user sees words appear immediately:

stream = client.responses.create(
    model="my-gpt",
    input="Write a short paragraph about the Nile.",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
print()

The stream is a series of events. You print the delta of each event whose type is response.output_text.delta, and ignore the rest. One subtlety worth knowing: if a failure happens in the middle of a stream, the HTTP status was already 200, so the error arrives as an event inside the stream rather than as a status code. Code that only checks the status will miss it.

Chat Completions, the older API. Most tutorials you will find use Chat Completions, and it is still supported, so recognise it:

completion = client.chat.completions.create(
    model="my-gpt",
    messages=[
        {"role": "developer", "content": "You are a concise assistant."},
        {"role": "user", "content": "Name three Azure regions in the Middle East."},
    ],
    max_completion_tokens=500,
)
print(completion.choices[0].message.content)

Here the conversation is a list of messages, each with a role. The instruction message takes the role developer, which behaves the same as the older system role; send one or the other, not both. The reply text is at completion.choices[0].message.content. Note max_completion_tokens rather than max_tokens: reasoning models reject max_tokens. Use Chat Completions when you must follow an existing tutorial or codebase, and Responses for new work.

Embeddings. An embedding turns text into a list of numbers so that texts with similar meaning end up close together. You use them for search, recommendations and retrieval. You need a separate deployment of an embedding model, such as text-embedding-3-small, and you call it by that deployment name:

result = client.embeddings.create(
    model="my-embedding",  # the name of YOUR embedding deployment
    input=["How do I reset my password?", "Forgot login credentials"],
)
print(len(result.data), len(result.data[0].embedding))

You get one vector per input. For text-embedding-3-small each vector has 1,536 numbers, and text-embedding-3-large produces 3,072. A single call accepts up to 2,048 inputs and 300,000 tokens in total, and each input may be up to 8,192 tokens. The two vectors above would be close to each other, which is the point. One rule matters for planning: you cannot mix vectors from different embedding models, and you cannot upgrade in place. If you change the model, you must embed everything again, so choose deliberately before you embed a million documents. When you are ready to store and search vectors, the guides on pgvector, Qdrant and FAISS pick up where this section stops.

Reasoning models and the parameters that change

Some deployments are reasoning models: they think through a problem internally before answering, which improves results on logic, maths and code at the cost of time and tokens. Models in the o-series and the GPT-5 family behave this way. For a beginner they matter because they accept different parameters from classic chat models, and sending the old ones produces a 400 error.

Three rules cover most of the trouble. First, use max_output_tokens on the Responses API, or max_completion_tokens on Chat Completions, never max_tokens. Second, do not send sampling controls such as temperature, top_p, presence_penalty, frequency_penalty, logprobs or logit_bias to the o-series and older GPT-5 models; they are rejected. The newest GPT-6 models are an exception and do accept logprobs, temperature and top_p. Third, the output budget includes the model's hidden reasoning tokens, which are billed as output tokens. A small max_output_tokens can be consumed entirely by thinking, leaving a response that is cut off or empty.

Reasoning models also take a reasoning effort, which says how hard to think:

response = client.responses.create(
    model="my-gpt",
    input="A train leaves at 09:10 and arrives at 11:45. How long is the trip?",
    reasoning={"effort": "low"},
    max_output_tokens=2000,
)
print(response.output_text)

The accepted effort values depend on the model: none, minimal, low, medium, high, and for some newer models xhigh and max. Support differs, and one detail catches people who upgrade: gpt-5.1 defaults to none, so code written for an older reasoning model that relied on a default will behave differently. If quality matters, pass the effort explicitly. Higher effort costs more tokens and takes longer, so match it to the task: low for simple lookups and formatting, higher for puzzles and code.

When a Responses call returns status: "incomplete" with an incomplete_details.reason of max_output_tokens, the budget ran out before the answer finished. Raise max_output_tokens, shorten the prompt, or lower the effort. Reading status is a habit worth forming: a response with HTTP 200 is not always a finished response.

Safety filters, now called guardrails

Every Azure OpenAI deployment has a safety layer in front of the model and behind it. Microsoft's classifiers read the prompt before it reaches the model and read the completion before it reaches you. They score text in four categories (hate, sexual content, violence, and self-harm) at levels from safe through low and medium to high. Out of the box, content at medium severity or above is blocked. These used to be called content filters; the current name is guardrails, and in the portal they live under "Guardrails + controls". Optional extras exist, such as prompt-injection detection (Prompt Shields), protected-material detection, and PII detection.

The filters are attached to a deployment, and as a beginner you will meet them as behaviour rather than configuration. There are two ways they show up in your results. If the prompt is blocked, the call fails with HTTP 400:

{"error": {"message": "The response was filtered", "param": "prompt", "code": "content_filter", "status": 400}}

In Python this raises openai.BadRequestError, and you can detect it by checking the code:

import openai

try:
    response = client.responses.create(model="my-gpt", input=user_text)
except openai.BadRequestError as err:
    if getattr(err, "code", None) == "content_filter":
        print("That request was blocked by the content guardrails. Please rephrase.")
    else:
        raise

If the completion is filtered, the call succeeds with HTTP 200, but the output is empty or cut short and the finish reason is content_filter. On Chat Completions you read completion.choices[0].finish_reason. That second case is the dangerous one, because nothing raised an error. An application that does not check will show the user a blank answer.

So build two habits. Catch the 400 and tell the user kindly to rephrase. And always look at the finish reason or the response status rather than assuming that a 200 contains a usable answer. You cannot switch the filters off on your own; relaxing them requires an application to Microsoft for modified content filtering, and that is for specific approved use cases. For a beginner project, the defaults are what you will use, and they are a feature: they are part of why a company can put the model in front of customers.

Configuration, cost and good housekeeping

A handful of settings and habits keep a first project cheap and tidy.

Keep configuration in the environment. Your code needs three values: the endpoint, the deployment name, and (if you use a key) the key. Read them from environment variables so the same code runs on your laptop, in a test environment and in production. A small settings block at the top of your file is enough:

import os

ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"].rstrip("/")
DEPLOYMENT = os.environ.get("AZURE_OPENAI_DEPLOYMENT", "my-gpt")
BASE_URL = f"{ENDPOINT}/openai/v1/"

Using os.environ[...] for the endpoint makes the program fail loudly on a missing variable instead of sending a call to nowhere. Put local values in a file named .env or a shell profile, and add that file to .gitignore before your first commit. A key committed to a repository must be treated as leaked: rotate it. The resource has two keys, key1 and key2, precisely so that you can switch your applications to one and regenerate the other without downtime.

Understand what you are billed for. Standard deployments are billed per token, with separate prices for input tokens, cached input tokens and output tokens, and the price depends on the model. Reasoning tokens count as output. Every response tells you what it consumed in response.usage, so you can calculate the cost of a call and watch it. Two rules reduce a bill quickly. Send less: trim long histories and unnecessary context. And set a sensible max_output_tokens so one runaway answer cannot cost much. Current prices change, so use the official pricing page rather than any figure from a guide.

Watch your limits. The service returns headers on each response, including x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-limit-tokens and x-ratelimit-remaining-tokens. On a 429 it adds retry-after-ms, the delay it wants you to wait. The OpenAI Python client already retries rate-limit errors and transient failures twice by default. You can raise that:

client = OpenAI(
    base_url=BASE_URL,
    api_key=token_provider,
    max_retries=5,
)

If you add your own retry library such as tenacity on top, set max_retries=0 on the client, or the two layers multiply and you send far more traffic than you expect, which makes the rate limit worse.

Keep an eye on the lifecycle. Models retire. Check the retirement schedule for your model when you start a project and again every few months. A deployment can be set to upgrade automatically to a newer default version of the same model, or to stay put; with the second setting, the deployment stops working on the retirement date. Subscribe to Azure Service Health advisories so that the notice reaches a person and not only an unread mailbox.

Common errors and how to read them

Azure OpenAI errors are specific once you know how to read them. The error body is JSON with a code and a message, and the HTTP status tells you which side to look at: 400 means your request was wrong, 401 and 403 mean identity or permission, 404 means something named does not exist, 429 means a limit, and 5xx means the service. The most common ones follow.

404 DeploymentNotFound: The API deployment for this resource does not exist. Three causes, in order of likelihood. You wrote the model name where the deployment name belongs. You spelled the deployment name with different capitalisation. Or the endpoint belongs to a different resource from the one where you created the deployment. The message also suggests waiting five minutes if you created the deployment very recently, and that is real. Check with az cognitiveservices account deployment list and compare the name character by character.

401 Unauthorized with an Entra token. The resource has no custom subdomain, or you called a regional endpoint instead of https://<name>.openai.azure.com. Recreate with --custom-domain, or add a subdomain, and use your own endpoint.

403 Forbidden or PermissionDenied. You authenticated, but lack a data-plane role. Assign Cognitive Services OpenAI User (or Foundry User on a project) on the resource, and wait up to five minutes. Being an Owner or Contributor does not count, because those roles are control-plane only.

DefaultAzureCredential failed to retrieve a token. The credential chain found nothing it could use. Run az login again, with --tenant if you belong to several tenants, then retry the az account get-access-token check from earlier. If the code works on your laptop and fails in Azure, the cause is almost always that the managed identity is not enabled, or that the role was given to your user account and not to the managed identity. With a user-assigned managed identity you must also tell the library which one by setting AZURE_CLIENT_ID.

429 rate limit. The messages read "Requests to the ... have exceeded call rate limit" or "Rate limit is exceeded". You used more tokens or requests per minute than the deployment's allocation. Back off using retry-after-ms, lower max_output_tokens, raise the deployment's capacity if you have spare quota, and smooth bursts, since the requests-per-minute limit is checked over windows of a second or ten. Two surprises: the service estimates tokens from your prompt plus max_output_tokens and not the tokens you will really use, so a very large output cap can trip the limit early; and failed requests still count toward the limit. A different 429 reads "The service is temporarily unable to process your request" or mentions high demand. That one is capacity pressure on Microsoft's side; wait and retry.

400 with content_filter. Covered earlier: a guardrail blocked the prompt. Rephrase it.

400 on a reasoning model naming temperature or max_tokens. You sent a parameter the model rejects. Remove the sampling parameters, and use max_output_tokens or max_completion_tokens. On GPT-5.6 models there is one more case: using tools with reasoning on Chat Completions fails with "Function tools with reasoning_effort are not supported ... To use function tools, use /v1/responses or set reasoning_effort to 'none'." The fix is the Responses API.

410 Gone. The model version is retired. Create a deployment of a current model and change the deployment name or its model.

status: "incomplete" with HTTP 200. The response ran out of max_output_tokens. Raise the limit, lower reasoning effort, or trim the prompt.

Putting it all together: a small command-line assistant

Here is one small project that uses what you have learned. It is a command-line assistant for a fictional help desk. It reads its settings from the environment, signs in with Entra ID, keeps a conversation going, streams the answer, handles filtered prompts and rate limits, and shows what each turn cost in tokens.

import os
import sys

import openai
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

ENDPOINT = os.environ["AZURE_OPENAI_ENDPOINT"].rstrip("/")
DEPLOYMENT = os.environ.get("AZURE_OPENAI_DEPLOYMENT", "my-gpt")

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
    base_url=f"{ENDPOINT}/openai/v1/",
    api_key=token_provider,
    max_retries=4,
)

INSTRUCTIONS = (
    "You are a friendly help-desk assistant for an IT team. "
    "Answer in at most four sentences. If you do not know, say so."
)

previous_id = None
print("Ask a question, or type 'quit'.")

while True:
    question = input("\nYou: ").strip()
    if question.lower() in {"quit", "exit"}:
        break
    if not question:
        continue

    try:
        stream = client.responses.create(
            model=DEPLOYMENT,
            instructions=INSTRUCTIONS,
            input=question,
            previous_response_id=previous_id,
            max_output_tokens=800,
            stream=True,
        )
        print("Assistant: ", end="", flush=True)
        for event in stream:
            if event.type == "response.output_text.delta":
                print(event.delta, end="", flush=True)
            elif event.type == "response.completed":
                previous_id = event.response.id
                usage = event.response.usage
                print(f"\n[tokens in={usage.input_tokens} out={usage.output_tokens}]")
    except openai.BadRequestError as err:
        if getattr(err, "code", None) == "content_filter":
            print("Assistant: I cannot help with that request. Please rephrase it.")
        else:
            print(f"Request problem: {err}")
    except openai.RateLimitError:
        print("Assistant: The service is busy. Please try again in a moment.")
    except openai.AuthenticationError:
        sys.exit("Sign-in failed. Run 'az login' and check your role assignment.")
    except openai.NotFoundError:
        sys.exit("Deployment not found. Check AZURE_OPENAI_DEPLOYMENT and your endpoint.")

Read it as a checklist of what you now understand. The settings come from the environment. The client signs in with DefaultAzureCredential, so there is no key in the file. The conversation continues through previous_response_id, which the service honours because responses are stored by default. The answer streams, and the final event gives you the usage numbers. Each exception class corresponds to an error from the previous section, and each produces a message a human can act on. The max_retries=4 setting means the client already retries transient failures before you ever see a RateLimitError.

Run it with:

export AZURE_OPENAI_ENDPOINT="https://MyOpenAIResource.openai.azure.com"
export AZURE_OPENAI_DEPLOYMENT="my-gpt"
python assistant.py

Ask it a few questions, then ask one that depends on the previous answer ("and how long does that take?") to confirm the conversation chaining works. Stop with quit. When you are finished with the project, remove everything with az group delete --name OAIResourceGroup --yes.

What you can now do, and what comes next

You can now explain, in plain terms, what Azure OpenAI is: a model hosted in your own Azure subscription, reached through an endpoint that you control and secure with Azure roles. You can name its nouns (resource, deployment, quota, token), and you know the difference that trips everyone up, that code addresses the deployment name and not the model name. You can create a resource group, a resource and a deployment from the command line, call the deployment with curl and with the Python openai client against the v1 API, sign in with Entra ID instead of a key, stream and chain responses, create embeddings, and read the common errors by status code and message.

You also know what you do not yet know. You have not sized capacity for real traffic or compared provisioned throughput with pay-as-you-go. You have not used structured outputs, tools, vision or batch jobs. You have not locked the resource to a private network, turned off keys, written infrastructure as code, or built dashboards and alerts. You have not planned a model upgrade across a retirement date. Those are the mid-level and senior topics, and the mid-level guide picks up exactly where you now stand.

Some neighbouring guides in this catalogue complete the picture. If you want to compare Azure's hosting with the original, read the OpenAI API guide; the calls are nearly identical and the differences are the ones you have just learned. If your team uses Azure for training and model management, the Azure ML guide covers that side of the platform. To put several model providers behind a single interface, see LiteLLM. And once your application is running, Langfuse shows how to trace what it actually sends and receives.

Sources

• Azure OpenAI overview: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/overview
• v1 API and API lifecycle: https://learn.microsoft.com/en-us/azure/foundry/openai/api-version-lifecycle
• REST API reference: https://learn.microsoft.com/en-us/azure/foundry/openai/reference
• Create a resource and deploy a model: https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/create-resource
• Responses API: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses
• Reasoning models: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning
• Deployment types: https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/deployment-types
• Quotas and limits: https://learn.microsoft.com/en-us/azure/foundry/openai/quotas-limits
• Manage quota and rate limits: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/quota
• Microsoft Entra ID and managed identity: https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/managed-identity
• Role-based access control for Azure OpenAI: https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/role-based-access-control
• Model retirement schedule: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirement-schedule
• Content filtering (guardrails): https://learn.microsoft.com/en-us/azure/foundry-classic/foundry-models/concepts/content-filter
• Data, privacy and security: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy
• Azure CLI installation: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli
• Azure OpenAI pricing: https://azure.microsoft.com/pricing/details/azure-openai/

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

Azure OpenAI is Microsoft's hosting of OpenAI models inside a customer's own Azure subscription. You create a resource, which owns an endpoint, then create a deployment, which is a named instance of a model with a type and a capacity, and your code calls that deployment by name. Access is controlled with Azure roles through Microsoft Entra ID, data commitments are contractual, and capacity is a token-per-minute quota.

Then the line that shows you have used it: "the model field takes the deployment name, not the model name, and a 404 DeploymentNotFound usually means I mixed the two up."

The model

The API in one screen

Know this call well enough to write it without looking. It is the current v1 pattern:

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
)
r = client.responses.create(model="my-gpt", input="This is a test.")
print(r.output_text)

Be ready to explain each line. base_url ends in /openai/v1/; the v1 API removed monthly dated api-version strings. model holds the deployment name. The Responses API is the recommended one, and Chat Completions still works with messages and max_completion_tokens. The older AzureOpenAI client with api_version is the legacy style.

Commands to have ready

az login
az account show --query "{subscription:name, tenant:tenantId}" --output table
az group create --name OAIResourceGroup --location eastus
az cognitiveservices account create --name MyOpenAIResource \
  --resource-group OAIResourceGroup --location eastus --kind OpenAI \
  --sku s0 --custom-domain MyOpenAIResource --yes
az cognitiveservices account deployment create --name MyOpenAIResource \
  --resource-group OAIResourceGroup --deployment-name my-gpt \
  --model-name gpt-5.1 --model-version "2025-11-13" --model-format OpenAI \
  --sku-name GlobalStandard --sku-capacity 10
az cognitiveservices account deployment list --name MyOpenAIResource \
  --resource-group OAIResourceGroup --output table
az account get-access-token --resource https://ai.azure.com --query expiresOn --output tsv

The last command is the quickest way to separate "I am not signed in" from "I lack a role". In an interview, saying why you pass --custom-domain and why capacity is in units of 1,000 tokens per minute earns more than reciting flags.

Keys versus Entra ID

Interviewers like this topic because it separates people who have only pasted a key from people who have thought about access.

An API key sent in the api-key header grants full data-plane access with no role check, and anyone holding it can use your quota. Entra ID sends a short-lived bearer token, and Azure checks a role. In Python you pass a token provider as api_key:

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
                api_key=token_provider)

The client calls the function before each request, so tokens refresh automatically. Entra needs a custom subdomain and a role: Cognitive Services OpenAI User on the resource, or Foundry User on a Foundry project. Owner and Contributor are control-plane roles and do not allow inference through Entra. Assignments take up to five minutes to propagate.

The classic traps and why they happen

The model name versus the deployment name. The service routes by deployment. A name that matches a model but not a deployment returns 404 DeploymentNotFound. Names are case-sensitive, and a deployment created in the last few minutes may not exist yet.

403 as an Owner. Owner and Contributor manage the resource; they do not call the model. Inference needs a data-plane role.

401 with a token. Either the resource has no custom subdomain, or the call went to a regional endpoint instead of the resource's own.

429 while "under quota". Rate limiting uses an estimate that includes max_output_tokens, counts rejected requests, and checks request rates over windows of a second or ten. A large output cap can trip it early, and a retry loop makes it worse.

Reasoning-model parameters. Reasoning models reject max_tokens and sampling controls like temperature. Use max_output_tokens (Responses) or max_completion_tokens (Chat Completions). Hidden reasoning tokens count toward the budget and are billed as output.

A 200 that is not an answer. A filtered completion arrives with HTTP 200 and finish reason content_filter. A response with status: "incomplete" ran out of output tokens. Check both.

Stale models. Models retire: GA models are deprecated at twelve months and retired at eighteen. A retired model returns 410 Gone.

Streaming errors. A failure in the middle of a stream arrives as an event inside an HTTP 200 stream.

Common interview questions

What is Azure OpenAI, and how does it differ from calling OpenAI directly? The models are the same family, but Azure OpenAI runs inside your Azure subscription. The differences are operational: the endpoint belongs to you, access is governed by Entra ID and Azure roles, billing is on your Azure account, networking can be restricted, and Microsoft's data commitments apply: prompts and completions are not available to other customers or to OpenAI and are not used to train the foundation models. The API shape is nearly the same, and the main code difference is that you address a deployment, not a model.

What is a deployment? A named instance of a model version on a resource, with a deployment type and a capacity. I choose the name. Code puts it in the model field. I can have several deployments of the same model with different capacities or guardrail settings, up to 32 standard deployments on a resource.

What does a resource contain? An endpoint, two keys, network settings, an optional managed identity and a custom subdomain. It lives in one region. It is of kind OpenAI, or AIServices if it is a Foundry resource, which adds other providers' models and projects.

Which deployment type would you start with, and why? Global Standard. It has the newest models, the lowest price and the highest default quota. I would move to Data Zone or regional Standard only if a residency rule required it, accepting that newer models arrive there later.

Explain data location. Processing location depends on the deployment type: global means any Azure region, data zone means within the US, the EU or Asia-Pacific, and standard means the resource's geography. Data at rest, such as stored files, stays in the resource's geography in every case. For a Gulf or Egyptian employer with a residency rule, I would read both statements carefully and confirm which models are available in the permitted region.

What is quota, and what happens when I exceed it? Quota is a per-subscription allowance of tokens per minute, per model and deployment type. I assign a slice to each deployment, and one unit of capacity is 1,000 TPM for standard deployments. Exceeding the deployment's rate limit returns 429. The response includes retry-after-ms. The client library retries twice by default, and I can raise that with max_retries.

How would you authenticate an application to Azure OpenAI? With Entra ID. Locally, DefaultAzureCredential uses my az login; in Azure, it uses a managed identity. I grant Cognitive Services OpenAI User on the resource. I avoid keys because a key gives full access with no role check and can leak. If I must use one, it is in an environment variable or Key Vault and never in code.

What is the Responses API, and how does it differ from Chat Completions? Responses is the recommended API for Azure OpenAI models. It takes input and optional instructions, supports chaining with previous_response_id, stores state for 30 days by default (disable with store=False), and returns text through output_text. Chat Completions takes a list of messages, returns text at choices[0].message.content, and is still supported. Chat Completions is also the syntax for calling other providers' models on the same endpoint.

How do you keep a conversation going? Models are stateless. Either chain with previous_response_id, or keep the history in my own list and send it every turn. Each turn re-sends the whole history, so cost per message grows with the conversation.

What is the Assistants API, and should I use it? It was an older stateful API with assistants, threads and runs. It was retired on 26 August 2026, so I would not use it. The Responses API, with conversations and tools, replaced it.

What is a reasoning model, and what changes in the request? A model that thinks before answering. I use max_output_tokens or max_completion_tokens, drop temperature and similar controls (except on GPT-6 models, which accept some), and may set reasoning={"effort": "low"}. Reasoning tokens are billed as output. gpt-5.1 defaults to effort none, so upgraded code must set it explicitly.

What are embeddings, and what is the main planning constraint? Vectors that represent meaning, so similar text is close together. They need an embedding deployment. text-embedding-3-small produces 1,536 numbers per input and text-embedding-3-large 3,072. You cannot mix vectors from different models or upgrade in place; changing the model means re-embedding everything.

What are guardrails? Safety classifiers on prompts and completions for hate, sexual content, violence and self-harm, at safe, low, medium and high severity, with medium and above blocked by default. They were called content filters. A blocked prompt returns 400 with code content_filter. A blocked completion returns 200 with finish reason content_filter, so I must check it.

You get 404 DeploymentNotFound. What do you check? In order: the model value equals the deployment name exactly, including case; the endpoint belongs to the resource that holds the deployment; and whether the deployment is very new. I confirm with az cognitiveservices account deployment list.

You get 403 after assigning a role. Why? Propagation can take up to five minutes. If it persists, I check that the role is a data-plane role such as Cognitive Services OpenAI User, that it is assigned at the resource scope to the identity actually making the call, and, in Azure, that I assigned it to the managed identity rather than to myself.

What happens when a model is retired? Calls return 410 Gone. GA models are deprecated at twelve months and retired at eighteen, with at least sixty days' notice. Deployments can upgrade automatically to a new default version or stay pinned; a pinned deployment stops working at retirement. I check the retirement schedule and follow Service Health notices.

What do you do if the same code works locally and fails in Azure? Usually identity. DefaultAzureCredential used my az login locally. In Azure it needs a managed identity enabled, with a role assigned to that identity, and AZURE_CLIENT_ID set for a user-assigned one.

How do you keep costs under control as a beginner? Small capacity, short experiments, a sensible output cap, trimmed histories, and deleting the resource group afterwards. I read usage on responses to compute cost per call, and I know reasoning tokens count as output.

What changed recently that a candidate should know? The v1 API replaced dated api-version strings, the Responses API became the recommended one, the Entra scope is https://ai.azure.com/.default, content filters became guardrails, the platform is now called Foundry, the Assistants API was retired, and quota moved to tiers and subscription-level pooling.

Why would a company pick Azure OpenAI over a public API key? Governance, mostly. The model sits behind the same identity, billing, networking and policy tools as the rest of the company's Azure estate. Security teams can say who may call it, auditors can see where it runs, and procurement already has the contract. The trade-off is extra setup: a resource, a deployment, quota and roles before the first call, plus the fact that new models may reach regional deployment types later than global ones.

How do you stream a response, and what is the catch? Pass stream=True and iterate over events, printing event.delta when the type is response.output_text.delta. The catch is that the HTTP status is 200 before the answer starts, so a failure midway arrives as an error event in the stream. Code that only checks the status will miss it, and the user sees a half-finished sentence.

What would you check before putting a first prototype in front of colleagues? That authentication uses a role rather than a shared key, that the output cap and the history length are bounded, that filtered and incomplete responses are handled with a friendly message, that retries sit in one layer, and that the model I chose is not close to retirement. I would also confirm where the data is processed and stored, and read the data-privacy page so that I can answer the first question the security team will ask.

How do you explain tokens to a non-technical manager? Tokens are the units the model reads and writes, roughly three quarters of a word in English. We pay by them and limits are set in them, so a long prompt, a long history or a long answer all cost more. Arabic text often uses more tokens per word than English, so I would test cost with real Arabic samples before promising a budget.

What is the difference between the control plane and the data plane? The control plane is Azure Resource Manager, where I create the resource, deployments, quota and policies, and it is what the az cognitiveservices commands use. The data plane is the model API at my endpoint, where prompts are sent. Roles are separate for each, which is why an Owner can still be refused when calling the model.

What a strong answer covers that a weak one misses

A weak answer describes Azure OpenAI as "OpenAI on Azure" and stops. A strong answer names what changes: deployments instead of model names, quota in tokens per minute, identity through Entra ID, and data location split between processing and storage.

A weak answer on authentication says "I use the key from the portal." A strong one says why a key is risky, names the role, and mentions the custom subdomain.

A weak answer on errors lists status codes. A strong one reads the code and message, reproduces with curl, and compares names, because most beginner failures are two names that differ.

A weak answer treats 200 as success. A strong one checks status and finish_reason for filtered or truncated output.

A weak answer uses examples with retired models. A strong one picks a durable model, calls the deployment name a placeholder, and mentions the retirement schedule.

When you do not know something, say how you would find out. "I would check the quota page in the portal and the model retirement schedule" is a better answer than a guessed number, because quotas, prices and model lists change.

Sixty-second self-test

Answer out loud without looking above.

1. What goes in the model field, and what happens if you put a model name there?
2. Name the five nouns from subscription to token.
3. Which deployment type do you start with, and why?
4. Which role lets you call the model with Entra ID, and why is Owner not enough?
5. What is the Entra scope for the data plane today?
6. What does a 429 mean, and what do you do first?
7. What two ways can a guardrail show up, and which one raises no error?
8. Which parameters do reasoning models reject?
9. How do you continue a conversation with the Responses API?
10. Why can you not mix embeddings from two models?

If you hesitated on more than two, reread the matching section of the detailed guide and run its Try it task. For the next step, see the OpenAI API guide for the original API and Azure ML for Azure's training and model-management side.

Part one of three. Most beginner problems with Azure OpenAI come from three confusions: the name of a model versus the name of its deployment, the control plane versus the data plane, and a successful HTTP status versus a usable answer. Start with the table, then the habits and the checklist.

Common errors at this level

The mistakes beginners actually make

Putting the model name in the model field

Symptom. Your first call returns 404 DeploymentNotFound: The API deployment for this resource does not exist.

Cause. You wrote gpt-5.1, the catalogue name. The service looks for a deployment with that name. If you named your deployment my-gpt, nothing matches.

Fix. Run az cognitiveservices account deployment list --name <resource> --resource-group <group> --output table, copy the deployment name exactly, and keep it in an environment variable. Names are case-sensitive.

Assuming Owner means you can call the model

Symptom. You own the subscription and still get 403 Forbidden or PermissionDenied from the data plane with an Entra token.

Cause. Owner and Contributor are control-plane roles. They create resources, deployments and keys. Inference with Entra ID needs a data-plane role. Even Cognitive Services Contributor does not allow it.

Fix. Assign Cognitive Services OpenAI User on the resource (or Foundry User on a Foundry project) to the identity that makes the call, then wait up to five minutes.

Creating the resource without a custom subdomain

Symptom. Key-based calls work, and Entra calls return 401. Batch jobs misbehave later.

Cause. Entra sign-in requires a custom subdomain, which you set with --custom-domain at creation. Without it the endpoint is a regional one that does not accept Entra tokens.

Fix. Always pass --custom-domain with the same name as the resource, and call https://<name>.openai.azure.com.

Committing a key

Symptom. A key appears in a repository, a screenshot or a chat message.

Cause. The key was pasted into code. A key grants full data-plane access with no role check, so whoever finds it can spend your quota.

Fix. Regenerate the key at once; the resource has key1 and key2 so you can rotate without downtime. Move the value to an environment variable, add your local env file to .gitignore, and switch to DefaultAzureCredential so the next version has no secret at all.

Sending parameters a reasoning model rejects

Symptom. 400, with a message naming max_tokens or temperature.

Cause. Reasoning models, including the o-series and the older GPT-5 models, do not accept max_tokens, temperature, top_p, presence_penalty, frequency_penalty, logprobs or logit_bias. The newest GPT-6 models are an exception and accept logprobs, temperature and top_p.

Fix. Use max_output_tokens on the Responses API or max_completion_tokens on Chat Completions, and remove the sampling parameters. Add optional parameters one at a time so the error tells you which one the model refuses.

Setting a tiny output cap on a reasoning model

Symptom. The call returns 200 but the text is empty, or the response has status: "incomplete" with incomplete_details.reason equal to max_output_tokens.

Cause. Hidden reasoning tokens count toward the output budget. A cap of 20 may be spent entirely on thinking.

Fix. Raise max_output_tokens, lower the reasoning effort, or shorten the prompt. Always check status before trusting output_text.

Treating HTTP 200 as success

Symptom. Users see a blank answer, and your logs show no errors.

Cause. A completion blocked by a guardrail returns 200 with finish reason content_filter. A mid-stream failure also arrives inside an HTTP 200 stream, as an error event.

Fix. Check finish_reason on Chat Completions and status on Responses. In streaming code, handle error events and not only response.output_text.delta.

Hammering the service and multiplying retries

Symptom. A loop produces a wall of 429 errors, and adding a retry library makes it worse.

Cause. The client library already retries rate-limit errors and transient failures twice. If you wrap it in another retry layer, the two multiply. Failed requests also count toward the limit, so retrying instantly deepens the hole.

Fix. Honour retry-after-ms, add jitter, and choose one layer. If you use a library such as tenacity, set max_retries=0 on the client. Raise the library's own retries with OpenAI(..., max_retries=5) if you prefer to rely on it.

Following a tutorial for the Assistants API

Symptom. Code that creates an assistant or a thread fails, or the documentation page you found does not match anything in the portal.

Cause. The Assistants API was retired on 26 August 2026. Tutorials written before then still circulate.

Fix. Use the Responses API. Its conversations and tools replace threads and runs. When a tutorial mentions AzureOpenAI with an api_version, treat it as dated and convert to OpenAI with the /openai/v1/ base URL.

Choosing a model that is about to retire

Symptom. A project that worked starts failing with 410 Gone, or a deployment cannot be created because the model is closed to new customers.

Cause. Models have lifecycles: GA models are deprecated for new customers at twelve months and retired at eighteen. Older families such as GPT-4o and the o-series have retirement dates in late 2026 and 2027.

Fix. Check the retirement schedule before you start, prefer a model with a long runway, and subscribe to Azure Service Health notices.

Mixing embeddings from different models

Symptom. Search returns nonsense after you upgrade or switch the embedding deployment.

Cause. Vectors from different embedding models live in different spaces. They cannot be compared, and you cannot upgrade a model in place.

Fix. Decide on the embedding model before you embed at scale. If you change it, re-embed everything. Record the model name next to every stored vector.

Forgetting the cost of long conversations

Symptom. A chat feature gets more expensive and slower the longer people use it.

Cause. Each turn sends the whole history as input tokens, and you are billed for them every time. Chaining with previous_response_id stores the history on the service, but the model still reads it.

Fix. Trim or summarise old turns, set an output cap, and read response.usage to see what each turn costs.

Leaving resources running

Symptom. A surprise on the bill from a forgotten experiment.

Cause. The resource, deployments and anything attached remain after you stop using them, and a stray script can call the model.

Fix. Put each experiment in its own resource group and delete the group when done: az group delete --name <group> --yes. Remove deployments first if you want quota released cleanly.

Habits worth adopting on day one

Make the smallest possible call first. Before you build anything, run one curl against /openai/v1/responses with a deployment name and a short input. If that works, the endpoint, the key or token, and the deployment are all correct, and any later failure is in your code. If it fails, you have removed your application from the question.

Name things so that mistakes surface early. Use clearly different names for the resource, the deployment and the model, and put the project and environment in the resource-group name. A deployment called my-gpt makes the model-versus-deployment confusion obvious, and a group called support-bot-dev tells you what to delete.

Keep configuration in the environment. The endpoint, the deployment name and any key belong in environment variables read at start-up. Read the endpoint with os.environ["..."] so a missing value fails loudly. The same code then runs on your laptop and in production with no edits.

Sign in with Entra ID from the start. Even on a laptop, DefaultAzureCredential plus a role costs about the same effort as a key, and you will never have to remove a secret from your code later. Use the scope https://ai.azure.com/.default, and check it with az account get-access-token --resource https://ai.azure.com.

Read the response, not only the text. Look at status, finish_reason and usage as a matter of routine. Those three fields tell you whether the answer finished, whether a guardrail intervened, and what it cost.

Write instructions as if for a new colleague. Say who the assistant is, who it is talking to, how long answers should be, and what to do when it does not know. "Answer only from the provided text; otherwise say you do not know" prevents more bad answers than any parameter.

Respect rate limits politely. Use retry-after-ms, add jitter, and keep one retry layer. Lower max_output_tokens to what you need. A smaller output cap helps both cost and rate limits, because the limiter uses an estimate that includes it.

Choose durable models for examples and learning. Pick something with a long runway, keep the deployment name in configuration, and treat model names in tutorials as examples. Check the quota page and the retirement schedule before a project, not after.

Keep the data question in view. Know the processing location of your deployment type and the storage location of your resource, and write both down. If your employer has a residency rule, confirm it before you send real data, and read Microsoft's data-privacy page for what abuse monitoring may store.

Track what you spend. Print token usage during development. Learn what each kind of call costs before you put it behind a button that a thousand people can press.

Learn the neighbours. When you outgrow single calls, the Langfuse guide covers tracing what your application sends, LiteLLM covers routing across providers, and pgvector covers storing the embeddings you create.

Reading an error in four steps

Beginners often paste an error into a search box and read ten forum posts. A faster method uses the error itself, because Azure OpenAI errors are specific.

First, read the HTTP status, since it tells you which side to look at. A 400 means the request is wrong: a bad parameter, a blocked prompt, or a context that is too long. A 401 or 403 is about identity or roles. A 404 means a named thing does not exist, nearly always the deployment. A 429 is a limit. A 410 is a retired model. Anything in the 5xx range is the service, and a retry after a short wait is reasonable.

Second, read the code and the message. The message often names the parameter or the fix. The deployment error even tells you to wait five minutes if the deployment is new, and the reasoning-model error names the parameter it rejected.

Third, reproduce the failure with the smallest curl call you can. If curl works and your program does not, the cause is in your program, usually a name, a scope or a library version. If curl fails the same way, the cause is in Azure: the deployment, the role, the endpoint or the quota.

Fourth, compare names character by character. Deployment name, endpoint, subdomain, scope and role are all strings that must match something elsewhere, and a surprising share of beginner problems are a name in one place that differs from a name in another.

Keep a note of each error you solve, with the status, the message and the fix. In a few weeks it becomes your own troubleshooting guide, and writing it forces you to understand the cause rather than only the cure.

Practice cards

A short checklist

Before you call the model for the first time:

1. az account show prints the subscription you intend to use.
2. The resource was created with --custom-domain.
3. You can state the deployment name, and it is not the model name.
4. The deployment type is GlobalStandard unless a residency rule says otherwise.
5. Your identity has Cognitive Services OpenAI User on the resource, or you are using a key held only in an environment variable.
6. A single curl call to /openai/v1/responses succeeds.

Before you share the code with anyone:

1. No key, token or endpoint secret is in the repository, and your env file is ignored.
2. The client uses the /openai/v1/ base URL and the Responses API.
3. Reasoning-model parameters are correct: max_output_tokens, no sampling controls.
4. The code checks status or finish_reason, and catches 400 with content_filter.
5. Retries are in one layer, and retry-after-ms is respected.
6. You know the retirement date of the model you chose.

Before you walk away:

1. You printed response.usage and know what a call costs.
2. The experiment lives in its own resource group.
3. You deleted what you no longer need with az group delete.
4. You wrote down one thing you would do differently, which is the best preparation for the mid-level guide.

This is part two of three. It picks up where Beginner ended: you have a resource, a deployment, and a working call through the v1 API. Here we take that to the level where the service behaves predictably in production, which means understanding how a request is actually routed, why a 429 arrives when your dashboard says you are under quota, and what happens to your code when Microsoft retires a model underneath it.

Where this picks up

Each section says what problem it solves before showing commands, and ends with a Try it you can run against a real resource in a few minutes. Model names change quickly on this platform, so every example uses a placeholder deployment name such as my-gpt. Replace it with the name of a deployment you own, and check the retirement schedule before you copy a model name into a pipeline.

How a request is routed and metered

Most mid-level confusion about Azure OpenAI comes from a wrong picture of what you are calling. You are not calling a model. You are calling a deployment, a named instance of a model and version that lives on a resource, with a deployment type (the SKU) and a capacity. The model field of every request carries the deployment name. That is the most important difference from the OpenAI API itself, and it explains the most common error in the service, 404 DeploymentNotFound. The message says the deployment does not exist, and it is literal: the name is case-sensitive, it must belong to the resource whose endpoint you are calling, and a deployment created in the last few minutes may not be visible yet.

A resource is an Azure Resource Manager object of type Microsoft.CognitiveServices/accounts. It comes in two kinds. kind: OpenAI is the classic Azure OpenAI resource, which exposes only the /openai/v1 surface. kind: AIServices is a Foundry resource, a superset that adds other providers' models, projects, and the agent service. Foundry resources expose three hostnames built from the custom domain: {name}.openai.azure.com for the OpenAI-compatible API, {name}.services.ai.azure.com for Foundry APIs and projects, and {name}.cognitiveservices.azure.com. For plain inference with the OpenAI SDK, you call the first one.

Two planes exist, and mixing them up wastes hours. The control plane is Azure Resource Manager at management.azure.com: creating accounts and deployments, reading quota, attaching guardrail policies. The data plane is /openai/v1/...: responses, embeddings, files, batches, fine-tuning, evals. Each plane has its own token audience, and a token for one is rejected by the other. That is why an engineer with Owner on the subscription can still get 403 when calling the model: Owner is a control-plane role.

Three things happen between your client and the model, and each can reject you for a different reason. The deployment layer checks the name and enforces its rate limit. The guardrail layer scores your prompt and, afterward, the output. Only then does the model run. A 404 is the first layer, a 429 is the rate-limit part of the first layer, and a 400 with code content_filter is the second. Knowing which layer produced an error tells you where to look.

Where the request is processed depends on the deployment type, not on the resource region. A Global deployment may process in any Azure region. A Data Zone deployment processes within the US, the EU, or APAC. Standard and Regional Provisioned deployments process within the Azure geography. Data at rest, which means files, stored responses and fine-tuned models, stays in the resource's geography regardless. For a Gulf or Egyptian employer with residency rules, that distinction is the first design question: a Global deployment is fine for public data and a poor fit for anything covered by a residency commitment. Data Zone covers the case where "EU" or "US" is acceptable, and regional types are the strictest option, though they receive new models last. The rollout order for new models is Global Standard, then Global Provisioned, then Data Zone, and finally regional types with no guaranteed date.

One more concept belongs here. The custom subdomain of the resource, the {name} in {name}.openai.azure.com, is required for Entra authentication. Pass --custom-domain when you create a resource from the CLI, because a resource without a proper subdomain fails Entra calls with 401 and has been reported to misbehave with Global Batch.

Authentication and RBAC in depth

Beginner showed DefaultAzureCredential and a bearer-token provider. The mid-level question is what that token is allowed to do, because the role model has several traps.

The data-plane audience is https://ai.azure.com/.default. Older documentation uses https://cognitiveservices.azure.com/.default, so you will see both in code you inherit. The control plane uses https://management.azure.com/.default. From a shell, az account get-access-token --resource https://ai.azure.com gives you a data-plane token for debugging with curl.

import os
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)

client = OpenAI(
    base_url=f"https://{os.environ['AOAI_RESOURCE']}.openai.azure.com/openai/v1/",
    api_key=token_provider,   # a callable: called before every request
)

Passing a callable as api_key is what makes this production-grade. The client invokes it before each request, and the azure-identity cache refreshes the token before it expires, so a long-running worker never holds a stale token.

The roles are where engineers lose time. Inference with Entra needs a data-plane role on the resource. Cognitive Services OpenAI User can call models and use the playground but cannot read keys or create deployments. Cognitive Services OpenAI Contributor adds deployments, fine-tuning, stored completions and On Your Data sources. Cognitive Services Contributor is a control-plane role that can read keys and manage guardrails, and it cannot make Entra inference calls. On a Foundry project, the equivalent least-privilege role is Foundry User. Role assignments take up to five minutes to propagate, so a 403 right after you assign a role is not always a mistake.

The Foundry roles were renamed, and the documentation advises using role GUIDs in code while the rename rolls out. This matters for Terraform and Bicep, where a name lookup that worked last quarter can fail.

For compute running in Azure, use a managed identity. A system-assigned identity needs no configuration beyond assigning it the role. With a user-assigned identity, DefaultAzureCredential will not guess which one to use, so set AZURE_CLIENT_ID or construct ManagedIdentityCredential(client_id=...). The classic "works on my laptop, fails in Azure" is one of three things: the managed identity is not enabled, the role was assigned to your user instead of the identity, or the client ID is missing for a user-assigned identity. For pipelines outside Azure, use a service principal or workload identity federation rather than a stored secret. If you must keep a key, store it in Key Vault and rotate between key1 and key2. Neighbouring guides worth reading here are Vault for secrets handling and Azure ML for the platform side of the same tenant.

The Responses API beyond hello world

The Responses API is the recommended surface for Azure OpenAI models. Chat Completions still works, and remains the syntax for calling other providers' models through /openai/v1/chat/completions, but new features land on Responses first. The mental shift from chat completions is that a response is a stored object, not just a return value.

By default the service retains response state for 30 days. That retention is what enables previous_response_id: instead of resending the whole conversation, you pass the ID of the last response and the service reconstructs the context. It is convenient and it has a cost in data governance, because your prompts now sit in the resource's geography for a month. If you need statelessness, pass store=False and manage history yourself by appending response.output items to your next input list. Retrieval and deletion are available through client.responses.retrieve(id) and client.responses.delete(id).

first = client.responses.create(model="my-gpt", input="Name three Gulf data-centre regions.")
follow = client.responses.create(
    model="my-gpt",
    previous_response_id=first.id,
    input="Which of those is nearest Cairo?",
)
print(follow.output_text)

# Stateless variant: you own the history, nothing is stored.
history = [{"role": "user", "content": "Summarise our SLA in two lines."}]
r = client.responses.create(model="my-gpt", input=history, store=False)
history += r.output
history.append({"role": "user", "content": "Now in Arabic."})

Structured outputs are the single most useful production feature, because they turn "parse the model's prose" into "validate a schema". In Responses you set text={"format": {"type": "json_schema", ...}}; in Chat Completions the parameter is response_format. A schema-constrained call fails loudly rather than returning something almost-JSON, and that is what you want at a system boundary.

schema = {
    "type": "json_schema",
    "name": "ticket",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "category": {"type": "string", "enum": ["billing", "outage", "other"]},
            "urgent": {"type": "boolean"},
        },
        "required": ["category", "urgent"],
        "additionalProperties": False,
    },
}
r = client.responses.create(
    model="my-gpt",
    input="Our payments page has been down since 9am.",
    text={"format": schema},
)

Function tools use tools=[{"type": "function", ...}] in Responses. The model does not run your function. It returns a tool call, your code runs it, and you send the result back in a follow-up request. Two limits to design around: tool definitions count against the context window (a GPT-4.1 known issue produces 400 context_length_exceeded when tool definitions alone pass 300K tokens), and a single tool description over 1,048,576 characters fails with string_above_max_length. Parallel tool calls are not supported when reasoning_effort is minimal. Other built-in tools exist, among them code_interpreter ({"type": "code_interpreter", "container": {"type": "auto"}}), file search, web search, MCP and image generation.

Vision and documents use input items. An image goes in as input_image with a URL, base64 data or a file ID, in PNG, JPEG, WebP or non-animated GIF form, up to 50 images and 50 MB per request. A PDF goes in as input_file with a file_data data URL. When you upload files, use purpose="assistants", since user_data is not supported.

Long-running agent loops eventually hit the context window. Responses offers compaction: automatic through context_management=[{"type": "compaction", "compact_threshold": 200000}], or manual with client.responses.compact(model=..., input=[...]). Use automatic compaction for agents you do not babysit, and manual compaction when you want to inspect what gets dropped.

Streaming has its own trap. You pass stream=True and handle events such as response.output_text.delta. If something fails midstream, the HTTP status was already 200, so the error arrives as an error event inside the stream, with types such as server_error, too_many_requests, forbidden and user_error. Code that only watches the status code will miss it.

Reasoning models and their parameter rules

Reasoning models spend hidden tokens thinking before they answer. Those tokens are billed as output tokens, they count toward your output limit, and you do not see them unless you ask for a summary. That one fact drives most of the surprises at this level: bills that look wrong, empty answers, and requests rejected for parameters that worked on older models.

The parameter rules differ by API. In Chat Completions the budget is max_completion_tokens, and max_tokens is rejected on reasoning models. In Responses it is max_output_tokens. Either way the budget covers reasoning plus visible output. On o-series and older GPT-5 models, temperature, top_p, presence_penalty, frequency_penalty, logprobs, top_logprobs, logit_bias and max_tokens are unsupported. GPT-6 models are the exception: they support logprobs, temperature and top_p. A developer role message behaves like system; send one or the other, never both.

reasoning_effort accepts none, minimal, low, medium, high, xhigh and max, but which values a model accepts varies:

• max works on GPT-6 and GPT-5.6, in the Responses API only.
• xhigh works on GPT-6, GPT-5.6, GPT-5.5, GPT-5.4 and gpt-5.1-codex-max.
• minimal is original GPT-5 only; it fails on gpt-5.1 and later, and on gpt-5-codex.
• gpt-5-pro supports only high, and gpt-5.1-codex-max does not support none.
• o1-mini does not support reasoning_effort at all.

The default is the second trap. gpt-5.1 defaults to none, so code migrated from an older reasoning model that relied on a default of medium will suddenly think less. Set the effort explicitly in every call you care about. Responses takes reasoning={"effort": "medium", "summary": "auto"}, and also exposes text={"verbosity": "low"} to control answer length independently of thinking depth. Newer controls include reasoning.mode (standard, or pro on GPT-6 and GPT-5.6 in Responses) and reasoning.context (auto, current_turn, all_turns).

r = client.responses.create(
    model="my-reasoning-deployment",
    input="Plan a zero-downtime migration of a 2 TB Postgres database.",
    reasoning={"effort": "high", "summary": "auto"},
    text={"verbosity": "low"},
    max_output_tokens=12000,
)
print(r.status, r.usage.output_tokens)

There is also a breaking change worth knowing by heart. On GPT-5.6 models, Chat Completions with tools combined with reasoning fails with Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'. It fires even when you never sent reasoning_effort, because the default is medium. The fix is to move that call to Responses, or to send reasoning_effort: "none". If you have a thin wrapper that hides which API you use, this is the case for migrating it.

Quota, rate limits and the real meaning of a 429

Quota is an allowance of tokens per minute (TPM), scoped per subscription, per model and per deployment type, that you carve up among deployments. For standard deployments one capacity unit is 1,000 TPM. A request-per-minute limit is derived from it: older chat models get 6 RPM per 1K TPM, and most newer ones get 1 RPM per 1K TPM. The limit is enforced per deployment, so splitting a workload across two deployments of the same model gives each its own bucket, which is a legitimate way to isolate a noisy batch from an interactive path.

Two pieces of the mechanism explain why 429 often appears when your metrics say you are under quota. First, the rate limiter works from an estimate, not billed tokens: prompt characters, plus max_tokens (or the equivalent output cap), plus best_of. A request with a generous output cap reserves that whole amount for the estimate even if the real answer is short. Second, RPM is evaluated over one- or ten-second windows, so a deployment with 600 RPM is throttled above 10 requests in any single second, even if the minute total is far below 600. Failed requests count toward the limit as well.

The practical consequences: set the output cap to what you actually need rather than the maximum; smooth bursts with a client-side queue or semaphore; and spread load across deployments when one tenant or feature should not starve another.

Not every 429 means the same thing, and the fix differs:

The response headers worth logging are x-ratelimit-limit-requests, x-ratelimit-limit-tokens, x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, the two reset headers, and retry-after-ms on 429.

Retries are where teams double their own load. The OpenAI Python SDK retries 429 and transient errors by default with max_retries=2. Raise it with OpenAI(..., max_retries=5) or per call with client.with_options(max_retries=8). If you add a library such as tenacity with its own exponential backoff, set max_retries=0 on the client, otherwise the two layers multiply and one logical failure becomes dozens of requests that all count against the limit.

from openai import OpenAI
from tenacity import retry, wait_random_exponential, stop_after_attempt

client = OpenAI(base_url=BASE_URL, api_key=token_provider, max_retries=0)

@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def ask(prompt: str):
    return client.responses.create(model="my-gpt", input=prompt, max_output_tokens=400)

Quota itself has been reshaped recently. Quota tiers (Free Tier 0 and Tiers 1 to 6) replaced the old Default and Enterprise levels. Tiers upgrade automatically based on usage, enterprise agreement status and payment history, and there is a preview policy, tierUpgradePolicy: "NoAutoUpgrade", to opt out. After May 7, 2026, subscription-level quota pooling began rolling out: for Global Standard, one pool per model and version is shared across regions, and Data Zone Standard shares one pool per zone. Do not assume your quota is per region on models where pooling has reached you. Check the current state rather than trusting an old runbook.

To inspect quota from automation, use the usages API, which needs the Cognitive Services Usages Reader role at subscription scope:

SUB=$(az account show --query id -o tsv)
LOC=eastus
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$LOC/usages?api-version=2024-10-01"

Quota names look like OpenAI.Standard.gpt-4o, in units of thousands of TPM. To ask whether capacity exists before you create a deployment, call the Model Capacities API at .../providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=<m>&modelVersion=<v>, which returns availableCapacity per location and SKU. Having quota does not guarantee capacity, and that second API is how you tell the two apart. Quota changes in the portal can take about 15 minutes, and increases are requested through https://aka.ms/oai/stuquotarequest.

Choosing a deployment type on purpose

A deployment type is a decision about price, latency guarantees, data location and commitment. Beginners take the default. At this level you choose deliberately, because the choice is hard to change under load.

Global Standard is the sensible start: newest models, lowest price, highest default quota, widest region support. Data Zone Standard keeps processing inside the US, the EU or APAC. Regional Standard keeps it in one geography and tends to get new models last. These are all pay-as-you-go, billed per input, cached input and output token, with reasoning tokens billed as output.

When you need steady throughput and predictable latency, there is provisioned throughput: you reserve capacity measured in PTUs, billed hourly or through one-month or one-year reservations, whether or not you use it. The order of operations matters. Check capacity first, create the deployment, and only then buy a reservation, because a reservation does not guarantee capacity. Provisioned deployments are not auto-upgraded when models retire; you migrate them yourself, either in place or side by side. When utilization (the metric AzureOpenAIProvisionedManagedUtilizationV2) reaches 100%, you get a 429.

Spillover is the safety valve. A provisioned deployment can overflow to a standard deployment of the same model and version in the same resource on 429, long-context 400, 500 or 503. Set the property spilloverDeploymentName on the deployment to apply it to all traffic, or send the header x-ms-spillover-deployment: <std-deployment> per request; the deployment property wins if both exist. Response headers x-ms-spillover-from-deployment, x-ms-deployment-name and x-ms-spillover-error tell you what happened, and the metric dimension IsSpillover lets you chart it. Spillover traffic is billed at standard rates, which is the cost of not dropping requests.

Three request-level service tiers sit on top, selected with service_tier:

• Priority (service_tier: "priority") is pay-as-you-go with a latency target and a premium rate. It is available on Global Standard and Data Zone Standard (US only, not the EU zone, not regional) for model versions 2025-12-01 and later. A ramp limit applies: growing TPM by more than 50% in under 15 minutes can downgrade requests to standard, and the response then reports service_tier: "default", billed at standard.
• Flex (service_tier: "flex", preview) is half price for delay-tolerant online work. Since 2026-09-25, unsupported models return HTTP 400 invalid_request_error and there is no automatic fallback; the first supported model is gpt-5.6-sol on Global Standard. Use a long client timeout of around 900 seconds, fall back explicitly to default on a 429, and monitor with the ServiceTierRequest dimension.
• Batch is offline work at half price, covered in the next section.

The header x-ms-service-tier overrides the body, with valid values flex, priority, default and auto. Remember the platform's header limit: at most ten custom headers pass through, and more returns HTTP 431. Microsoft has also said future versions will not pass custom headers through, so do not build a design that depends on arbitrary ones.

Batch jobs from end to end

Batch is the cheapest way to process large volumes when nobody is waiting: a 24-hour target at 50% off, running against a separate enqueued token quota, so it does not compete with your interactive traffic. You deploy a Global Batch (or Data Zone Batch) deployment, and dynamic quota, which lets a deployment use spare capacity when available, is recommended for it.

The flow has four steps. First, write a JSONL file in which each line is a complete request. The same deployment name must appear on every line.

{"custom_id": "task-0", "method": "POST", "url": "/v1/responses", "body": {"model": "my-batch-deployment", "input": "Classify: invoice overdue"}}
{"custom_id": "task-1", "method": "POST", "url": "/v1/responses", "body": {"model": "my-batch-deployment", "input": "Classify: login fails"}}

Second, upload it with purpose="batch". You can set an expiry of 14 to 30 days, expressed in seconds (1209600 to 2592000). Third, create the job with completion_window="24h", the only supported value. Fourth, poll and collect.

f = client.files.create(
    file=open("tasks.jsonl", "rb"),
    purpose="batch",
    extra_body={"expires_after": {"seconds": 1209600, "anchor": "created_at"}},
)
job = client.batches.create(
    input_file_id=f.id,
    endpoint="/v1/responses",
    completion_window="24h",
)

import time
while job.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(60)               # poll no more often than every 60 seconds
    job = client.batches.retrieve(job.id)

if job.status == "completed":
    out = client.files.content(job.output_file_id).text

The statuses you will see are validating, failed, in_progress, finalizing, completed, expired, cancelling and cancelled. A job that completes successfully still has an error_file_id that points at a zero-byte file, which surprises people who treat the presence of that field as failure. Instead, read the output file and match results to inputs by custom_id.

Most batch failures happen at validation and have a recognisable code in errors.data[].code: invalid_json_line, too_many_tasks (over 100,000 requests), url_mismatch, model_not_found, duplicate_custom_id, empty_file, model_mismatch, invalid_request (the deployment must be a global batch type), input_modified and input_no_permissions. One failure has a misleading message: "Validation failed: A valid model deployment name couldn't be extracted from the input file" is typically a UTF-8 file with a byte order mark. Save the file as plain UTF-8 without a BOM. Windows tools produce these files regularly.

The limits to design around are 100,000 requests per file, 200 MB per input file (1 GB with bring-your-own storage), and 500 input files without an expiry set, rising to 10,000 with one. Job records are kept for 365 days, and output files do not expire by default, so set output_expires_after if the data is sensitive. Batch does not support Assistants or On Your Data.

Guardrails: handling what the filter does

Content filters are now called guardrails, and in the portal they live under "Guardrails + controls". They are classifiers applied to the prompt and to the completion for hate, sexual content, violence and self-harm, at severities safe, low, medium and high. The default blocks medium and above. Optional additions include Prompt Shields (against user prompt attacks and indirect attacks hidden in documents), protected-material detection for text and code, groundedness, PII, task adherence and custom blocklists. A guardrail is configured per deployment; in Azure Resource Manager it is a raiPolicies object attached through the deployment's raiPolicyName.

Your code meets guardrails in three distinct ways, and each needs handling.

A blocked prompt returns HTTP 400 with code: "content_filter" and the message "The response was filtered". In Python that is a BadRequestError whose error.code is content_filter. Catch it and show the user a useful message rather than a stack trace.

A blocked completion returns HTTP 200 with finish_reason: "content_filter" and empty or partial output. Because the status is success, only code that checks finish_reason notices. Treat that check as mandatory in every Chat Completions handler.

A filter failure is the quiet one. When the filtering system itself is unavailable, the request can complete unfiltered, and the annotation shows content_filter_results.error.code: "content_filter_error". Log it and alert on it, because it means the control you rely on was not applied.

from openai import BadRequestError

try:
    r = client.responses.create(model="my-gpt", input=user_text)
except BadRequestError as e:
    if getattr(e, "code", None) == "content_filter":
        reply = "That request was blocked by our content policy."
    else:
        raise

Annotations tell you which category fired and at what severity. In Responses they appear in response.model_extra["content_filters"]; in Chat Completions they are in prompt_filter_results and content_filter_results. Log them as structured fields. A rising count of low-severity annotations is an early signal that users are probing, long before blocks begin.

Turning filters off or to annotate-only on completions requires approval for modified content filtering, and that is a process, not a setting. Separately, Azure runs abuse monitoring, which may store flagged prompts and completions within the geography for human review, and customers can apply to modify it. If approved, the resource shows ContentLogging: false. Do not confuse the two systems: guardrails act on a request in flight, and abuse monitoring is a retrospective control.

For indirect attacks, where untrusted text such as a web page or an email is fed into the prompt, the platform recommends formatting that marks document boundaries so that Prompt Shields can tell data from instructions. For application-layer validation that goes beyond the platform's classifiers, see Guardrails, and for testing how your prompts behave against adversarial input see Promptfoo.

Lifecycle: models retire, and your code has to survive it

On a managed platform the model under your deployment has a lifespan. The current policy: a GA model retires 18 months after launch and becomes Deprecated, meaning closed to new customers, at 12 months. A retired model returns 410 Gone. Preview models get at least 30 days' notice and are force-upgraded. GA retirement notices arrive at least 60 days ahead. Retirement dates cannot be extended. Anthropic, DeepSeek, Fireworks and Mistral GA models follow a 12-month lifecycle.

As of the schedule dated 2026-09-21, several dates are close enough to matter: gpt-4.1-nano retires on 2026-10-14, codex-mini on 2026-11-15, and the whole o-series (o1, o1-pro, o3, o3-pro, o3-deep-research, o3-mini, o4-mini) on 2026-11-19. gpt-4o 2024-05-13 retires 2026-12-09, and several others, among them gpt-4o-mini, gpt-4.1 and gpt-4.1-mini, on 2027-04-14. Treat this list as a snapshot and read the retirement schedule itself, which is linked in Sources.

You can query lifecycle state programmatically, which is how you build an alert instead of relying on someone reading an email:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$LOC/models?api-version=2024-10-01" \
  --query "value[?model.name=='gpt-4o'].{v:model.version,status:model.lifecycleStatus,dep:model.deprecation.inference}"

There is a naming gotcha in the response. The API value Deprecating corresponds to what the documentation calls Deprecated (still works), and the API value Deprecated means Retired. A script that alerts on the string "Deprecated" fires after the outage.

What happens at retirement is controlled per deployment by versionUpgradeOption: OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired or NoAutoUpgrade. With NoAutoUpgrade the deployment simply stops working when the version retires. Standard, Global Standard, Data Zone Standard and Priority auto-upgrade according to that option; provisioned and batch deployments need manual or side-by-side migration. Replacement models typically appear in Global Standard about 90 days before retirement, and in provisioned regions about 30 days before.

The part engineers underestimate is behavioural drift. An automatic upgrade swaps the model while keeping your deployment name, so your code sees no change and your outputs may. Two documented examples: gpt-5.1 defaults to reasoning_effort: none, and GPT-5.6 rejects function tools with reasoning on Chat Completions. Microsoft usually names the replacement about 90 to 120 days ahead, but do not wait for that announcement. Keep an evaluation set of real prompts and expected properties, run it against candidate models when a retirement lands on your calendar, and decide with data. See Promptfoo and RAGAS for ways to build that suite.

A related platform change deserves a section of its own in your runbook. Microsoft is auto-upgrading eligible Azure OpenAI resources to Foundry resources. An upgrade keeps the name, endpoint, keys, custom domain, network settings, fine-tunes, batches and stored completions, and rollback is supported. Scheduled resources show a foundryAutoUpgrade block in the ARM JSON at api-version 2026-01-15-preview, and a "Resource upgrade" blade in the portal. To opt out, set foundryAutoUpgrade: { mode: 'Disabled' } through Bicep or AzAPI, or use "Defer upgrade" in the portal. Private-network and customer-managed-key resources are not selected at first. For Terraform, the AzureRM provider needs version 4.56.0 or later to change kind from OpenAI to AIServices without destroying the resource.

Review role assignments before an upgrade. After it, the "Cognitive Services User" role grants all Foundry features, while "Cognitive Services OpenAI User" stays OpenAI-only. Wildcard roles and policies written for the old kind can behave differently. Finally, the Assistants API is retired as of 2026-08-26. If any of your code still uses threads and runs, the migration mapping is assistants to prompts or agents, threads to conversations, and runs to responses, on top of the Responses API.

Observability: metrics, logs and traces

Azure OpenAI emits platform metrics automatically at one-minute granularity, and they are the first place to look when something is slow or failing. The main one is AzureOpenAIRequests, with dimensions you will filter on constantly: StatusCode, ModelDeploymentName, ModelName, ModelVersion, StreamType, IsSpillover, ServiceTierRequest, ServiceTierResponse, ApiName, OperationName and Region. Token volume comes from ProcessedPromptTokens, GeneratedTokens and TokenTransaction. AzureOpenAIContextTokensCacheMatchRate shows how well prompt caching is working, which is a direct lever on both cost and latency.

Latency has several metrics, and picking the right one matters: AzureOpenAITimeToResponse is roughly time to first token, AzureOpenAITTLTInMS is time to last byte, AzureOpenAINormalizedTBTInMS is time between tokens, and AzureOpenAINormalizedTTFTInMS and AzureOpenAITokenPerSecond round out the set. Per the metrics page, tokens per second, time to response and time between tokens are not available for Standard deployments, so a missing chart is not necessarily a broken setup. For provisioned deployments, use AzureOpenAIProvisionedManagedUtilizationV2; the V1 metric is deprecated. For guardrails, RAIRejectedRequests, RAIHarmfulRequests and RAITotalRequests show the filter's activity.

Avoid the legacy Cognitive Services metrics (Latency, TotalCalls, ClientErrors, SuccessRate); the documentation warns that they mislead. Always read a latency metric next to a token-count metric.

Resource logs require a diagnostic setting; nothing is stored until you create one. The categories are Audit, RequestResponse, Trace, AzureOpenAIRequestUsage and ManagedNetworkEvent. They land in the AzureDiagnostics table, with metrics in AzureMetrics. Crucially, prompt and completion bodies are not in these logs. If you need them for debugging or audit, capture them at the API Management gateway with its LLM logging, or in your own application through tracing. That is also where you decide what to redact.

az monitor diagnostic-settings create \
  --name aoai-to-law \
  --resource "$(az cognitiveservices account show -n MyOpenAIResource -g OAIResourceGroup --query id -o tsv)" \
  --workspace "$LAW_ID" \
  --logs '[{"category":"RequestResponse","enabled":true},{"category":"Audit","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

AzureDiagnostics
| take 100
| project TimeGenerated, _ResourceId, Category, OperationName, DurationMs, ResultSignature, properties_s

Platform metrics tell you how the service is doing. They do not tell you what your application asked of it or whether answers were good. For that you need application-level tracing. OpenTelemetry instrumentation of your OpenAI calls gives you spans with model, token counts and latency per user request, which you can ship to Application Insights or to an LLM-specific tool. See OpenTelemetry, Langfuse, Prometheus and Grafana for the pieces. For alerting, Azure Monitor metric and log alerts work on all of the above, and a sensible starting set is: error rate by StatusCode, 429 rate by deployment, provisioned utilization above a threshold, spillover count, and RAIRejectedRequests spikes.

Infrastructure as code and the AI gateway

Click-built resources drift, and a deployment created by hand has no review trail. Both Bicep and Terraform handle Azure OpenAI well, with a caveat about preview properties.

In Bicep, the account is Microsoft.CognitiveServices/accounts@2025-06-01. The properties that matter for a secure setup are customSubDomainName, disableLocalAuth, publicNetworkAccess, and, for a Foundry resource, allowProjectManagement.

param name string
param location string = resourceGroup().location

resource account 'Microsoft.CognitiveServices/accounts@2025-06-01' = {
  name: name
  location: location
  kind: 'AIServices'
  sku: { name: 'S0' }
  identity: { type: 'SystemAssigned' }
  properties: {
    customSubDomainName: name
    disableLocalAuth: true
    publicNetworkAccess: 'Enabled'
    allowProjectManagement: true
  }
}

resource chat 'Microsoft.CognitiveServices/accounts/deployments@2025-06-01' = {
  parent: account
  name: 'my-gpt'
  sku: { name: 'GlobalStandard', capacity: 10 }
  properties: {
    model: { format: 'OpenAI', name: 'gpt-5.1', version: '2025-11-13' }
    versionUpgradeOption: 'OnceNewDefaultVersionAvailable'
  }
}

Deployments are child resources with sku.name and capacity, and properties.model carries format, name and version. They can also carry raiPolicyName for a guardrail and spilloverDeploymentName for provisioned overflow.

In Terraform, use azurerm_cognitive_account and azurerm_cognitive_deployment, version 4.56.0 or later for non-destructive kind changes, and the AzAPI provider for preview properties that AzureRM does not yet expose, such as foundryAutoUpgrade. See Terraform for state, modules and workflow. Two Azure Policy hooks are useful guardrails for a platform team: the field Microsoft.CognitiveServices/accounts/deployments/sku.name lets you restrict which deployment types teams may create, and a built-in policy exists to restrict allowed models.

Delete with care. Deleting a resource soft-deletes it, and if it still had deployments when deleted through the API or CLI, the quota stays locked for 48 hours unless you purge the deleted resource. Viewing and purging deleted accounts needs Contributor at subscription scope. In Terraform destroy-and-recreate workflows this surfaces as a mysterious quota error right after a clean destroy.

When several applications share a resource, put a gateway in front. Azure API Management's AI gateway adds the controls the service itself lacks: llm-token-limit enforces per-key TPM or token quotas, llm-emit-token-metric emits token counts with dimensions for chargeback, llm-semantic-cache-lookup and llm-semantic-cache-store provide semantic caching (requiring Azure Managed Redis or RediSearch plus an embeddings deployment), and llm-content-safety adds a content-safety check. Backends can be pooled with round-robin, weighted, priority or session-aware load balancing, with a circuit breaker that honours Retry-After. The gateway authenticates to the backend with its managed identity, so client applications never see a key. If you want a provider-neutral gateway in your own cluster instead, LiteLLM fills a similar role.

Continuous delivery for a service that calls Azure OpenAI should use workload identity federation rather than a stored secret. GitHub Actions can sign in to Azure with OIDC, and Kubernetes workloads can use a managed identity through workload identity so pods carry no keys at all. FastAPI is a common front for the service itself, where a single module-level client is created once and reused across requests.

Testing code that calls a remote, changing model

A model call is slow, costs money, and returns different text each time, so a naive test suite is either expensive or flaky. The answer is to test in layers and keep each layer honest about what it proves.

At the bottom, put your own logic behind a thin function: build the request, call the client, parse the result. Unit-test everything except the call by passing in a fake client. Because the OpenAI client is an ordinary object, you can substitute a stub that returns a canned response with status, output_text and usage, and you can make it raise RateLimitError or BadRequestError with code content_filter to prove your handlers behave. Those are the cases that are hard to produce on demand against the real service, and they are the ones that break in production.

class FakeResponses:
    def __init__(self, status="completed", text="ok"):
        self.status, self.text = status, text
    def create(self, **kw):
        class R: pass
        r = R(); r.status = self.status; r.output_text = self.text
        r.incomplete_details = None
        return r

class FakeClient:
    def __init__(self, **kw): self.responses = FakeResponses(**kw)

def test_incomplete_is_reported():
    out = answer(FakeClient(status="incomplete", text=""), "hi")
    assert out.error == "truncated"

In the middle, run contract tests against a real, small deployment on a schedule rather than on every commit: a handful of calls that assert shape, not wording. Does the structured-output schema still validate? Does the tool call round-trip? Does the parameter set you send still get accepted by the current model? These tests would have caught the GPT-5.6 tools change and the gpt-5.1 effort default, because both are about request acceptance, not answer quality.

At the top, run evaluations on a fixed dataset of real prompts, scored by rules, by a judge model, or by both. These are what you run when a retirement lands, when you change a prompt, or when you move to a cheaper model. Keep them out of the unit test path, and make the dataset boring and representative. The Evals API at /openai/v1/evals is now generally available, and the open tools in this catalogue, Promptfoo and RAGAS, run similar suites in your own pipeline.

Pin your client libraries. The v1 API removed the monthly api-version churn, but SDKs still move, and the Foundry SDK in particular has had version-coupling problems: if you see an AttributeError or ModuleNotFoundError from azure-ai-projects, check pip show azure-ai-projects and install >=2.3.0 alongside openai>=3.0.0. Treat an SDK bump as a change to test, not a background update.

Putting it all together

Here is how the pieces combine into a service you would be comfortable putting your name on. Picture an internal support-triage service for a company in Riyadh that must keep processing inside a defined region.

The infrastructure lives in Bicep: a Foundry resource with a custom subdomain, disableLocalAuth: true, a system-assigned identity, and two deployments. One is a Data Zone Standard deployment for interactive classification, with a guardrail policy attached and versionUpgradeOption set to OnceNewDefaultVersionAvailable. The second is a Data Zone Batch deployment, used overnight to backfill historical tickets at half price against the separate enqueued-token quota, so it never competes with live traffic. The application runs in Kubernetes with workload identity, and its identity holds only the least-privilege inference role on that resource.

The code uses one module-level OpenAI client with a bearer-token provider, max_retries set once and no second retry layer. Every call specifies an explicit reasoning effort and max_output_tokens, uses a JSON schema for structured output, and passes through a single function that checks status, finish_reason and BadRequestError with code content_filter. Tickets that come back filtered or incomplete are queued for a human, not dropped. A fake client covers those paths in unit tests, a nightly contract test runs a handful of real calls, and a fixed evaluation set runs whenever the model changes.

Operations are visible. A diagnostic setting sends RequestResponse and Audit logs to Log Analytics. Alerts watch 429 rate by deployment, error rate by StatusCode, and RAIRejectedRequests. Application traces carry token counts per ticket. A scheduled job queries the models API for lifecycleStatus and opens a ticket 90 days before any retirement date that touches a deployed model, remembering that the API value Deprecated means retired. Finally, a runbook records the Foundry auto-upgrade state, the role assignments to review before it, and who decides whether to defer.

Where you are now

You can predict how a request is routed and which layer produced an error. You know why a token's audience and a role's scope decide whether a call works, and why a key bypasses all of it. You use the Responses API with state, schemas and tools, and you know to check status before trusting output. You set reasoning effort and token budgets deliberately, and you know which parameters each model family rejects.

You can explain a 429 from the estimator and the one-to-ten-second windows, and keep your retry layers from multiplying. You choose between Global, Data Zone, provisioned, Priority, Flex and Batch on purpose, run a batch job end to end, handle blocked prompts, blocked answers and missing filters, and track model retirement before it becomes an outage. You provision with Bicep or Terraform, observe with metrics, logs and traces, and test without paying for every run.

The senior level is about operating this as a platform for other teams: provisioned throughput sizing, private networking, multi-tenant isolation, governance and incident playbooks.

Sources

• Azure OpenAI overview
• v1 API and API lifecycle
• REST API reference
• Responses API
• Reasoning models
• Quotas and limits
• Manage quota and rate limits
• Deployment types
• Provisioned throughput
• Spillover traffic management
• Priority processing
• Flex processing
• Batch
• Model retirement schedule
• Model lifecycle and support policy
• Upgrade Azure OpenAI to Foundry
• Automatic upgrades to Foundry
• RBAC for Foundry
• RBAC for Azure OpenAI
• Managed identity and Entra ID
• Content filtering and guardrails
• Monitoring Azure OpenAI
• Monitoring data reference
• Create a resource and deploy a model
• API Management AI gateway capabilities
• Data, privacy and security
• Azure OpenAI pricing

This file prepares you for the questions an interviewer asks someone who already runs Azure OpenAI at work. They are scenario and trade-off questions, so a memorised definition will not carry you. For each, the model answer shows the reasoning, followed by what separates a strong answer from a weak one. Say the answers aloud in your own words; the order of thought matters more than the exact phrasing.

How these interviews usually go

At this level an interviewer is testing three things. First, whether you can diagnose from symptoms: given an error or a metric, can you name the layer that failed? Second, whether you know the platform's sharp edges, the things documented in release notes rather than tutorials. Third, whether you make trade-offs with numbers and constraints instead of preferences. A candidate who answers "it depends" and then names what it depends on scores far higher than one who picks a favourite.

A reliable structure for any scenario answer is: restate the symptom, name the layers that could produce it, say which you check first, then say what you would change permanently so it does not recur.

Scenario questions

"Requests return 404 DeploymentNotFound, but the deployment shows in the portal. What do you check?"

The message means the endpoint you called has no deployment by that name. So I compare three things. First, the exact string in the model field against the deployment name, which is case-sensitive and is not the model name; a very common cause is sending gpt-5.1 when the deployment is called my-gpt. Second, whether the base URL belongs to the same resource as the deployment, since teams with a dev and a prod resource often point a config at the wrong one. Third, age: a deployment created in the last few minutes may not be visible yet, and the error text itself says so.

The permanent fix is to stop hard-coding names: read the deployment name from configuration per environment, and have a startup check that lists deployments or makes one cheap call, so a bad config fails at boot rather than on the first user request.

Strong answer: distinguishes deployment name from model name immediately, and mentions the second resource. Weak answer: "regenerate the key" or "the model is down".

"Everything works with your user, but the pod in Kubernetes gets 401 or 403."

I separate the two codes. 401 with an Entra token usually means the resource has no custom subdomain, or the pod is calling a regional endpoint instead of https://<name>.openai.azure.com. 403 means authentication succeeded and authorisation failed. For 403 I look at who the pod really is. Is a managed identity or workload identity actually attached? Was the role assigned to that identity rather than to my own user? If it is a user-assigned identity, is AZURE_CLIENT_ID set, because DefaultAzureCredential will not choose between identities? Then the role itself: Owner and Contributor are control-plane roles and do not allow inference, and Cognitive Services Contributor specifically cannot make Entra inference calls. The identity needs Cognitive Services OpenAI User, or Foundry User on a Foundry project. Finally, I allow for up to five minutes of propagation.

Strong answer: knows the control-plane versus data-plane split and names the specific role. Weak answer: "give it Owner", which both fails and violates least privilege.

"Latency is fine, but 429s appear while the metrics show usage well under quota."

This is the estimator at work. The rate limiter does not use billed tokens. It estimates from prompt characters plus the output cap (max_tokens or max_output_tokens) plus best_of. So a request that asks for a huge output reserves it all even if the real answer is short. In addition, request limits are evaluated over one- or ten-second windows, so a burst of eleven requests in one second can throttle a 600 RPM deployment while the minute average is low. Rejected requests count too, which makes a retry storm self-sustaining.

I would check the output cap first, because it is the cheapest to fix. Then I would look at how the client sends: is there a burst when a batch job starts? I would also count retry layers. The OpenAI SDK retries twice by default, and if the code also wraps calls in a retry library, one failure becomes many requests. The fixes are a realistic output cap, a client-side concurrency limit to smooth bursts, one retry layer with jitter that honours retry-after-ms, and separate deployments so an interactive path and a bulk path do not share a bucket.

Strong answer: mentions the estimator, the short windows, and retry multiplication. Weak answer: "ask for more quota" as the first and only move.

"A call that worked yesterday now returns 400 about max_tokens or temperature."

The likely cause is that the deployment now points at a reasoning model, or a deployment auto-upgraded to one. Reasoning models reject max_tokens, and on o-series and older GPT-5 they reject temperature, top_p, penalties and logprobs. The budget parameter becomes max_completion_tokens in Chat Completions or max_output_tokens in Responses, and it covers hidden reasoning as well as the visible answer. I would confirm what the deployment currently serves, since an upgrade keeps the name while changing the model, and then fix the request.

The deeper issue is that the code was coupled to model behaviour that nobody pinned. I would add a contract test that sends the real parameter set to a small deployment on a schedule, so that a model change surfaces as a failed test instead of a failed request.

Strong answer: names the upgrade-keeps-the-name behaviour and the contract test. Weak answer: removes parameters until it works and moves on.

"Responses return 200 but users see empty or cut-off answers."

A 200 does not prove an answer. With the Responses API I check status and incomplete_details. If the status is incomplete with reason max_output_tokens, the budget ran out, and on a reasoning model the likely culprit is that reasoning consumed most of it. I would raise the budget, lower the reasoning effort, or both, and measure the output tokens afterwards. In Chat Completions I check finish_reason: length means the same thing, and content_filter means the guardrail blocked the completion, which returns 200 with empty or partial output.

For streaming I also check for an error event inside the stream, because a midstream failure arrives after the 200 has already been sent. The permanent fix is a single response-handling function that every call goes through, which checks status, finish reason and filter annotations, and logs what it finds.

Strong answer: lists the three silent-failure shapes. Weak answer: blames the prompt.

"The business wants all processing to stay in the EU. What do you deploy?"

I would ask what exactly must stay in the EU: processing, storage, or both. Data at rest, meaning files, stored responses and fine-tunes, stays in the resource's geography regardless of deployment type. For processing, a Global deployment can run in any Azure region, so it is out. Data Zone Standard processes within the EU data zone, and regional Standard or Provisioned processes inside one geography. Data Zone is usually the best balance, since it has broader model availability than regional types, which receive new models last.

I would also check what that choice costs us: the newest model may not yet exist in the zone, the quota is pooled per zone, and Priority processing is not available in the EU data zone. I would state those trade-offs to the requester rather than discovering them in production. For the Gulf the same reasoning applies with the constraint names changed, and I would verify which zones and regions actually offer the model before promising anything.

Strong answer: separates processing from storage, and names a trade-off. Weak answer: "put it in a European region", which ignores deployment types.

"A model your product uses is being retired in 90 days. Walk me through it."

First I confirm exactly which deployments, versions and deployment types are affected, using the models API and each deployment's versionUpgradeOption. Standard types auto-upgrade according to that option, but provisioned and batch deployments need manual or side-by-side migration, so those are the high-risk ones. I check whether the option is NoAutoUpgrade, because then the deployment will simply start returning 410 Gone on the date.

Next I choose candidates and evaluate them on our own prompts rather than waiting for the vendor's recommended replacement. I compare quality, latency, cost and parameter compatibility, and I look at behaviour defaults that changed, such as gpt-5.1 defaulting to no reasoning. Then I roll out gradually: deploy the candidate under a new deployment name, shift a fraction of traffic, watch the metrics, and keep the old deployment until the date. Retirement dates cannot be extended, so there is no buffer to hope for.

The permanent fix is process: a scheduled query of lifecycle status that opens a ticket well ahead, and an evaluation set that exists before the next retirement is announced.

Strong answer: side-by-side rollout, own evaluation set, and the NoAutoUpgrade point. Weak answer: "we will switch to whatever Microsoft recommends."

"Your monthly bill doubled. Where do you look?"

I start with tokens, not requests: input, cached input and output, per deployment, from the metrics. Then I ask what changed. A new model with different pricing? A move to a reasoning model, where hidden reasoning is billed as output? A prompt that grew and crossed a long-context price threshold, which on GPT-5.6 and GPT-6 is decided by input tokens? A tier change, since Priority carries a premium? A retry storm doubling requests?

Once I know the driver I pick the lever. Smaller model if the evaluation allows it, explicit output caps, lower reasoning effort, stable prompt prefixes for caching, and Batch or Flex for work nobody is waiting on. For visibility I would use Cost Analysis filtered by the deployment billing tag, and token metrics per team through the gateway.

Strong answer: knows reasoning tokens bill as output and that long-context pricing depends on input size. Weak answer: "reduce the quota", which limits rate and not spend.

Trade-off questions

"Global Standard, Data Zone, or provisioned throughput?"

I start from constraints. If there is no residency requirement and the workload is variable, Global Standard gives the newest models, the lowest price and the highest default quota. If residency matters, Data Zone. If we need steady throughput or a latency guarantee and can show sustained utilization, provisioned throughput, since it bills hourly whether used or not and only pays off at high utilization. Provisioned deployments should have spillover to a standard deployment so peaks degrade gracefully instead of failing, and spillover traffic is billed at standard rates.

One caution: a reservation does not guarantee capacity, so I check capacity and create the deployment first.

Strong answer: ties the choice to utilization and mentions spillover and migration cost. Weak answer: "provisioned is faster, so use it", which is not the reason and wastes money.

"Batch, Flex, or Priority?"

It depends on who is waiting. Nobody waiting and a deadline of hours: Batch, at half price, with its own enqueued-token quota, so it does not disturb interactive traffic. Someone is waiting but can tolerate delay: Flex, also half price, but still a preview feature, supported on very few models, and it returns 400 on unsupported ones with no automatic fallback, so the code needs an explicit fall back to default on a 429. Someone is waiting and delay hurts: Priority, which costs a premium and has a ramp limit, because growing TPM by more than half in under fifteen minutes can downgrade requests to standard.

In an interview I would add that the choice can be per request through service_tier, so a single application can use more than one.

Strong answer: names the Flex limitations and the Priority ramp limit. Weak answer: treats them as interchangeable "cheaper" options.

"API key or Entra ID?"

Entra with managed identity wherever the caller runs in Azure, and workload identity federation for pipelines elsewhere. Keys grant full data-plane access with no role checks, they must be rotated, and they leak into logs and repositories. Entra enforces RBAC, supports per-identity audit, and needs no secret to manage. The costs are real but small: a custom subdomain is required, role propagation takes minutes, and local development needs az login. For a platform that wants this enforced, set disableLocalAuth: true, so that keys cannot be used even by mistake.

Strong answer: mentions disableLocalAuth and the no-role-check property of keys. Weak answer: "keys are simpler", with no mention of risk.

"Call Azure OpenAI directly, or through a gateway?"

Directly is simpler and has lower latency, and for one application with one team that is often right. A gateway earns its place when several applications share capacity, when you need per-team token limits and chargeback, when you want failover across deployments or regions, or when you want to keep keys away from clients. Azure API Management's AI gateway offers llm-token-limit, llm-emit-token-metric, semantic caching, and backend pools with a circuit breaker. A self-hosted option such as LiteLLM gives provider neutrality. The costs of a gateway are another component to run, another hop, and another place where a bug can sit, and semantic caching needs a Redis-style store and an embeddings deployment.

Strong answer: states the threshold at which a gateway becomes worth it. Weak answer: adds a gateway by default or refuses one on principle.

"Responses API or Chat Completions?"

Responses is the recommended API for Azure OpenAI models and is where new features land: stateful chaining, compaction, built-in tools, background mode. Chat Completions still works and remains the syntax for other providers' models on the same endpoint, and a large existing codebase may reasonably stay on it for now. The trade-off to name is statefulness: Responses retains state for 30 days by default, which is convenient and which has data-governance implications, so you pass store=False when you want stateless operation. And there is a concrete reason to move: on GPT-5.6, tools combined with reasoning on Chat Completions fails.

Strong answer: names the 30-day retention and the GPT-5.6 reason. Weak answer: "Responses is newer so it is better."

"Pin the model version, or let it upgrade?"

Letting Standard-type deployments follow the default version keeps you supported and on current models, but it changes behaviour without a code change. Pinning with NoAutoUpgrade gives stability, and the deployment stops working at retirement. My answer is to let non-critical paths follow the default and to pin critical paths only with a dated migration plan and an evaluation suite that lets me move deliberately before the retirement date. The principle is that pinning delays a decision and does not remove it.

Strong answer: says pinning only postpones the decision. Weak answer: pins everything and forgets.

Rapid-fire knowledge checks

An interviewer may drop short factual questions to calibrate you. Be ready with these.

• What is the data-plane token audience? https://ai.azure.com/.default, with older docs using https://cognitiveservices.azure.com/.default. The control plane is https://management.azure.com/.default.
• What does the model field hold? The deployment name, not the model name.
• Which role lets a user call the model with Entra? Cognitive Services OpenAI User, or Foundry User on a Foundry project. Cognitive Services Contributor cannot.
• Why does gpt-5.1 need an explicit effort? Its default is none.
• What does completion_window accept for Batch? Only "24h".
• How many custom headers pass through? Ten. More returns HTTP 431.
• What does 410 Gone mean? The model version is retired.
• Where are prompt and completion bodies logged? Nowhere in the platform's resource logs; use gateway logging or application tracing.
• What does the API value Deprecated mean in the models API? Retired. Deprecating means what the docs call Deprecated.
• How long is Responses state retained by default? Thirty days.
• What is the Assistants API status? Retired on 2026-08-26; migrate to Responses-based agents.

What a strong answer covers that a weak one misses

Strong candidates share a handful of habits. They name the layer before the fix: routing, authentication, rate limit, guardrail or model. They quote the specific mechanism, such as the estimator, the role scope, the status field or the default effort, instead of a general "it might be a quota issue". They separate the incident fix from the permanent fix. They put numbers on a trade-off: utilization for provisioned, tokens for cost, dates for retirement. And they admit what they would verify, for example "I would check the current retirement schedule, since these dates move", which reads as maturity and not as ignorance.

Weak answers reach for quota increases, Owner roles or key regeneration first, treat a 200 as success, and describe the platform as it was a year ago, with dated api-version strings or the Assistants API. Interviewers notice currency, because this service changes quickly.

Sixty-second self-test

1. Which three things can each return an error before the model runs?
2. Why can a 429 occur under quota?
3. Name two parameters a reasoning model rejects.
4. Which role lets Entra inference work, and which common role does not?
5. What does NoAutoUpgrade do at retirement?
6. Where do you find a filtered completion if the status is 200?
7. What is the cheapest option for a job nobody is waiting on?

If any answer is shaky, reread the matching section of the detailed file before the interview. The detailed guide covers routing, authentication, quota, deployment types, batch, guardrails, lifecycle, observability and testing in that order.

This file collects what pays off when you use Azure OpenAI at work: the errors you will meet, the habits ranked by leverage, practice cards to rehearse, and operational recipes you can paste into a runbook. Everything here assumes the v1 API and the Responses API. Model names are placeholders, because deployments and retirement dates change faster than any guide, so check the current schedule before you copy a name.

Errors you will actually meet

Each entry gives the symptom, the cause, and the fix. They are ordered roughly by how often they appear in support threads.

DeploymentNotFound

Symptom: 404 DeploymentNotFound: The API deployment for this resource does not exist.

Cause: the model field holds a name that is not a deployment on this resource. Usual culprits are the model name instead of the deployment name, different capitalisation, or an endpoint that belongs to another resource. A deployment created in the last few minutes can also be invisible briefly.

Fix: copy the exact name from az cognitiveservices account deployment list, load it from configuration, and verify the endpoint matches the resource. Add a startup check that fails fast.

401 with an Entra token

Symptom: a valid-looking bearer token is rejected with 401.

Cause: the resource has no custom subdomain, or the client is calling a regional endpoint.

Fix: configure a custom subdomain and call https://<name>.openai.azure.com/openai/v1/. Create resources with --custom-domain from the start so this never arises.

403 despite being Owner

Symptom: you can manage the resource but inference returns 403 or PermissionDenied.

Cause: Owner and Contributor are control-plane roles. Cognitive Services Contributor, despite the name, cannot make Entra inference calls. Or the role was assigned moments ago.

Fix: assign Cognitive Services OpenAI User (or Foundry User on a Foundry project) on the resource to the identity that makes the call, then wait up to five minutes.

Works locally, fails in Azure

Symptom: DefaultAzureCredential succeeds on your laptop and fails in the deployed service.

Cause: locally it uses your az login. In Azure it needs a managed identity that is enabled, holds the role, and, for a user-assigned identity, is named by AZURE_CLIENT_ID.

Fix: enable the identity, give the role to that identity rather than to yourself, and set the client ID.

429 that does not match your metrics

Symptom: throttling while usage looks low.

Cause: the limiter estimates tokens from prompt size plus the output cap, evaluates request rate over one- or ten-second windows, and counts rejected requests. A large max_output_tokens reserves capacity you never use.

Fix: lower the cap to what you need, smooth bursts with a concurrency limit, and keep one retry layer. Honour retry-after-ms.

429 with a temporarily lower limit

Symptom: x-ratelimit-limit-tokens is below the TPM you configured.

Cause: a temporary adjustment on the shared Standard pool.

Fix: back off, spread traffic across deployments or regions, and consider provisioned throughput if it recurs. It usually clears within hours.

Reasoning model rejects parameters

Symptom: 400 mentioning max_tokens or temperature.

Cause: reasoning models do not accept max_tokens, and o-series and older GPT-5 models do not accept sampling parameters either. GPT-6 models do accept temperature, top_p and logprobs.

Fix: use max_completion_tokens (Chat Completions) or max_output_tokens (Responses), and remove unsupported sampling parameters.

GPT-5.6 tools on Chat Completions

Symptom: Function tools with reasoning_effort are not supported for gpt-5.6-sol in /v1/chat/completions.

Cause: tools combined with reasoning on Chat Completions are not supported on GPT-5.6, and the default effort is medium even when you send nothing.

Fix: use the Responses API, or send reasoning_effort: "none".

200 with an incomplete response

Symptom: the call succeeds and output_text is empty or cut off; status is incomplete with reason max_output_tokens.

Cause: the output budget, which includes reasoning tokens, ran out.

Fix: raise the budget, lower the effort, shorten the prompt, and always check status first.

Filtered prompt or completion

Symptom: 400 with code content_filter ("The response was filtered"), or a 200 with finish_reason: "content_filter".

Cause: a guardrail threshold was crossed on the prompt or on the output.

Fix: catch BadRequestError and check the code, check finish_reason on every response, show a friendly message, and review the guardrail configuration if legitimate traffic is blocked.

431 Request Header Fields Too Large

Symptom: HTTP 431 on a request that is otherwise fine.

Cause: more than ten custom headers.

Fix: reduce them, and do not design around custom headers passing through, because Microsoft has said future versions will not pass them.

410 Gone

Symptom: a deployment that worked now returns 410 Gone.

Cause: its model version retired and the deployment was set to NoAutoUpgrade, or it is a provisioned or batch deployment that did not migrate.

Fix: create a deployment on a supported model, evaluate it, switch traffic, and put the retirement date in a calendar next time.

Flex returns 400

Symptom: invalid_request_error when sending service_tier: "flex".

Cause: since 2026-09-25 unsupported models return 400, with no fallback to Standard.

Fix: use a supported model, or send service_tier: "default".

Batch validation failures

Symptom: the job fails with "A valid model deployment name couldn't be extracted from the input file", or with a code such as duplicate_custom_id, model_mismatch or invalid_json_line.

Cause: commonly a byte order mark in the JSONL file, a different deployment name on some lines, or malformed lines.

Fix: save as plain UTF-8, use one Global Batch deployment name on every line, keep custom_id unique, and validate the file locally before upload. Remember that completion_window must be "24h".

Quota still locked after deleting a resource

Symptom: you deleted a resource and cannot recreate its deployments for lack of quota.

Cause: the resource was soft-deleted with deployments still present, and the quota stays locked for 48 hours.

Fix: purge the deleted resource, which needs Contributor at subscription scope. Delete deployments first in scripted teardown.

Foundry SDK import or attribute errors

Symptom: AttributeError or ModuleNotFoundError from azure-ai-projects.

Cause: version mismatch between azure-ai-projects and openai.

Fix: run pip show azure-ai-projects, then install "azure-ai-projects>=2.3.0" "openai>=3.0.0".

Practices ranked by leverage

These are in order of how much they prevent per hour spent. Do the first few before the rest.

1. Go keyless. Use Entra with a managed identity and set disableLocalAuth: true. It removes secret rotation, leak risk and the RBAC bypass in one change, and it is the highest-return security move available.
2. Put one function in front of every model call. It checks status, finish_reason, filter annotations and token usage, and it is the only place that sets retries and timeouts. Most production incidents are an unhandled response shape, and this function turns them into logged, handled events.
3. Set reasoning effort and output budget explicitly, always. Defaults differ across models and change across upgrades. An explicit setting is a form of documentation and a regression guard.
4. Keep an evaluation set of real prompts. Twenty to fifty representative cases with checkable expectations are enough to compare models, effort levels and prompts. It is what makes retirements and cost cuts safe.
5. Track retirement dates in your own system. Query the models API on a schedule, remember that the API value Deprecated means retired, and alert 90 days out.
6. Choose the deployment type per workload. Interactive on Standard or Priority, bulk on Batch, delay-tolerant online work on Flex where supported, steady high volume on provisioned with spillover.
7. Cap outputs to what you need. It lowers cost, lowers latency, and reduces the estimator's reservation, which lowers spurious 429 responses.
8. Create a diagnostic setting on day one. Logs are not retroactive. Without a setting, an incident has no history.
9. Define infrastructure in Bicep or Terraform. Quota, guardrails, deployments and network settings should be reviewable. Use AzAPI for preview properties.
10. Pin SDK versions and test upgrades. The v1 API ended api-version churn, but libraries still change.

Habits for safer prompts and requests

Treat the deployment name as configuration and never as a literal. Put it in an environment variable per environment, and name deployments by purpose, such as triage-fast and triage-deep, rather than by model. Then a model swap changes a deployment's contents and not your code.

Keep stable text at the start of prompts and variable text at the end. Prompt caching rewards a stable prefix, and AzureOpenAIContextTokensCacheMatchRate shows whether it is working. Put long, unchanging instructions first and the ticket text last.

Prefer structured outputs over parsing prose. A JSON schema with strict set and additionalProperties false turns a vague failure into a loud one. Validate the parsed result in your own code as well, since a schema guarantees shape and not truth.

Keep tool definitions small. They count against the context window, a single tool description has a length limit, and every tool you add is more for the model to misuse. Give each tool a narrow job and a short, precise description.

Mark untrusted text. When you place web pages, emails or user documents in a prompt, delimit them clearly, because the platform's indirect-attack detection works better when document boundaries are explicit. Never let model output flow unchecked into a shell, a database query or an HTTP call.

Practice cards

Work through these on a sandbox resource. Each takes ten to twenty minutes.

Operational recipes

Check who you are and that you can get a token

az login
az account show --query "{subscription:name, tenant:tenantId}" --output table
az account get-access-token --resource https://ai.azure.com --query expiresOn --output tsv

If the last command prints a timestamp, token acquisition works and any remaining failure is about roles or endpoints.

Create a resource and a deployment that will actually work

az cognitiveservices account create \
  --name MyOpenAIResource --resource-group OAIResourceGroup \
  --location eastus --kind OpenAI --sku s0 \
  --custom-domain MyOpenAIResource --yes

az cognitiveservices account deployment create \
  --name MyOpenAIResource --resource-group OAIResourceGroup \
  --deployment-name my-gpt \
  --model-name gpt-5.1 --model-version "2025-11-13" --model-format OpenAI \
  --sku-name GlobalStandard --sku-capacity 10

The --custom-domain flag is the one people forget. Without it, Entra calls fail with 401. For a Foundry resource, use --kind AIServices.

A minimal smoke test with curl

TOKEN=$(az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv)
curl -s -X POST "https://MyOpenAIResource.openai.azure.com/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"model":"my-gpt","input":"Reply with the single word: ready","max_output_tokens":50}'

Use this in a pipeline step after each deployment to prove the resource, role and deployment line up.

See what quota you have used

SUB=$(az account show --query id -o tsv)
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/eastus/usages?api-version=2024-10-01"

You need Cognitive Services Usages Reader at subscription scope. Quota names look like OpenAI.Standard.gpt-4o, and units are thousands of TPM.

Ask whether capacity exists before creating a deployment

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=gpt-5.1&modelVersion=2025-11-13"

The response lists availableCapacity per location and SKU. This is the check to run before buying a provisioned reservation.

Find models that are about to retire

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/eastus/models?api-version=2024-10-01" \
  --query "value[].{name:model.name,version:model.version,status:model.lifecycleStatus}" -o table

Read lifecycleStatus carefully. Deprecating is what the docs call Deprecated, and Deprecated is Retired.

Tell a client to retry sanely

client = OpenAI(base_url=BASE_URL, api_key=token_provider, max_retries=5)
resp = client.with_options(max_retries=8, timeout=900).responses.create(
    model="my-gpt", input=prompt, service_tier="flex", max_output_tokens=800,
)

Raise retries on the client for general use and per call for special cases, and use a long timeout for Flex. If you add your own retry library, set max_retries=0 so the layers do not multiply.

Handle every response shape in one place

from openai import BadRequestError

def ask(client, prompt, **kw):
    try:
        r = client.responses.create(model=DEPLOYMENT, input=prompt, **kw)
    except BadRequestError as e:
        if getattr(e, "code", None) == "content_filter":
            return {"ok": False, "reason": "blocked_prompt"}
        raise
    if r.status == "incomplete":
        return {"ok": False, "reason": r.incomplete_details.reason}
    return {"ok": True, "text": r.output_text, "tokens": r.usage.output_tokens}

Every caller then deals with one result shape, and the three silent failures become explicit values.

Turn on resource logs

az monitor diagnostic-settings create \
  --name aoai-to-law \
  --resource "$(az cognitiveservices account show -n MyOpenAIResource -g OAIResourceGroup --query id -o tsv)" \
  --workspace "$LAW_ID" \
  --logs '[{"category":"RequestResponse","enabled":true},{"category":"Audit","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Remember that prompt and completion bodies are not captured here. Add gateway logging or application tracing if you need them.

Alerts worth creating first

When something breaks at night

Keep a short triage order in the runbook so whoever is on call does not improvise. First, read the status code split in the AzureOpenAIRequests metric: a wall of 429 is capacity, a wall of 400 is a request or guardrail change, a wall of 401 or 403 is identity, and 5xx is the platform. Second, check what changed: a deployment version upgrade, a new release of your service, a role or network change, a Foundry upgrade on the resource. Third, check Azure Service Health for advisories under the service name Azure OpenAI Service. Only then start experimenting. Write down each step you take, because the notes become the post-incident review, and the review is where the permanent fix is decided.

A review checklist before you ship

Run through this before a service that calls Azure OpenAI goes live. Authentication is Entra-based with a managed identity, keys are disabled where possible, and the role is least-privilege. The deployment name comes from configuration. Reasoning effort and output budget are set on every call. One function handles status, finish reason and filter annotations. There is exactly one retry layer. Guardrails are attached to the deployment and the code handles blocked prompts and completions. A diagnostic setting exists. Alerts cover 429, errors and utilization. Infrastructure is defined in code. The retirement date of every model in use is recorded, with an owner. An evaluation set exists and has been run against the current model. For related practice, see Terraform, OpenTelemetry and Promptfoo.

What to read next

The senior files take this further into provisioned throughput sizing, spillover design, private networking, multi-tenant isolation and governance. Before you move on, make one real service in your own work pass the checklist above. The gap between knowing these practices and having applied them once is where most of the learning happens.

This is part three of three. It takes every topic from Beginner and Mid one level further, so that each one gains a dimension of ownership: who may do it, what it costs, what breaks at volume, and what you tell the other teams when it changes underneath them. It also adds the work that only exists once Azure OpenAI is a platform you run for other people, rather than an API you call.

Up to this point the questions were about making calls succeed. From here they change. Which of forty teams is consuming the shared quota? What happens to every deployment when a model is retired in ten weeks? Who can read prompts, who can read keys, and what does a leaked key allow? And when Microsoft changes the product underneath you, which of your assumptions quietly stops being true?

Where this picks up

A note on naming before anything else, because you will be asked. The product is Azure OpenAI in Microsoft Foundry Models, and the platform around it is Microsoft Foundry. It was previously called Azure AI Foundry, and before that Azure AI Studio. Content filters are now called guardrails. Documentation lives under learn.microsoft.com/en-us/azure/foundry/..., and older URLs redirect. This guide is written against the v1 API (generally available, opt-in since August 2025), openai 3.x for Python and Azure CLI 2.90, with facts checked in September 2026. Treat anything about models, quotas and prices as a snapshot, because those pages change monthly, and say so in your own runbooks.

I am starting with the shape of the thing you own, because every later decision depends on knowing which unit holds which property.

The units you own, and what each one holds

A beginner sees "an Azure OpenAI". A platform owner sees five distinct units, each of which holds a different set of properties and fails in a different way.

The subscription holds quota. This matters more than any other fact in the guide, because quota is the scarce resource and it is shared by everything in the subscription. Quota is assigned per subscription, per region (moving to per-subscription pooling, covered later), per model and per deployment type. Two teams in one subscription compete for one pool, which is either exactly what you want or a cause of incidents, depending on whether you decided it.

The resource holds identity and network. A resource is an ARM object of type Microsoft.CognitiveServices/accounts with kind: OpenAI (Azure OpenAI) or kind: AIServices (a Foundry resource, with allowProjectManagement: true). It carries the region, the custom subdomain, key1 and key2, the network rules, the managed identity and any customer-managed key. Per region per subscription there is a limit of 30 Azure OpenAI resources and 100 AIServices resources. Data at rest stays in the resource's geography regardless of the deployment type you pick.

The project exists only on Foundry resources. It is a sub-resource and an access-control boundary for agents, files and evaluations. Its endpoint is https://<resource>.services.ai.azure.com/api/projects/<project>.

The deployment is the unit your callers see, and the source of the most common beginner error: the model field holds the deployment name, never the model name. A resource can hold at most 32 standard deployments and 10 fine-tuned deployments.

The two planes have different audiences, and a token for one is rejected by the other. Control-plane calls (create a deployment, read quota, edit a guardrail policy) use the audience https://management.azure.com/.default. Data-plane calls (inference, files, batches, evals) use https://ai.azure.com/.default. Older documentation used https://cognitiveservices.azure.com/.default for the data plane, and you will meet it in code written before the change. Verify which your runtime issues:

# data plane token (inference)
az account get-access-token --resource https://ai.azure.com --query expiresOn --output tsv
# control plane token (ARM)
az account get-access-token --resource https://management.azure.com --query expiresOn --output tsv

The Foundry resource is a superset of the Azure OpenAI resource. It adds other providers' models, Agent Service, Foundry Tools and projects. An upgrade keeps the name, endpoint, keys, custom domain, network and identity settings, fine-tunes, batches and stored completions, and rollback is supported. We will return to the upgrade, because Microsoft is performing it on eligible resources whether or not you asked.

What breaks first: failure modes at scale

Platform ownership starts with a ranked list of what fails, because the order drives where you spend effort. For Azure OpenAI the order is surprisingly stable.

First, rate limiting that does not match your mental model. Rate limits are enforced per deployment. The token estimate used for admission is derived from prompt characters plus max_tokens (or max_output_tokens), plus best_of, and is not the billed count. A request that sets a generous maximum output reserves quota it will never use, so a fleet of services all asking for 4,096 output tokens while producing 200 will throttle at a fraction of the quota you think you have. RPM is evaluated over one-second or ten-second windows, so a deployment limited to 600 RPM throttles above ten requests in any single second, even when the minute looks quiet. Failed requests still count towards the limit. Your dashboards, which aggregate over a minute, show headroom while callers see 429.

There are three distinct kinds of 429, and the response text tells you which. "Requests to ... have been limited" or "Rate limit is exceeded" means your deployment's TPM or RPM. "The service is temporarily unable to process your request" or "System is experiencing high demand" means backend capacity pressure, which no quota change fixes. And a 429 whose x-ratelimit-limit-tokens header is below your configured TPM means the shared Standard pool has applied a temporary adjustment. Each of these has a different remedy, and a retry policy that treats them the same will either hammer a saturated backend or give up on a transient adjustment.

Second, the retry multiplication trap. The OpenAI Python SDK retries 429 and transient errors by default with max_retries=2. Teams then wrap calls in tenacity or a framework retry. Two layers of retry multiply: three attempts times six attempts is eighteen requests per logical call, each of which counts against the limit. At platform level you publish one rule: exactly one retry layer, owned by the client, with max_retries set deliberately and backoff that honours retry-after-ms. If an application uses its own library, it sets max_retries=0.

Third, region-wide events. A Global Standard or Data Zone Standard deployment routes to capacity across a wider area, but the deployment is still anchored to the resource's region. If the primary region has an outage, traffic routed there is affected. A single resource in a single region is therefore a single point of failure however it is deployed, and the answer is a design decision made before the incident: multiple resources in separate regions behind a gateway, with the failover behaviour tested.

Fourth, lifecycle failures. A retired model returns 410 Gone. A deployment with versionUpgradeOption set to NoAutoUpgrade stops working at retirement, by design. These do not appear as load problems at all. They appear as a sudden step in the error rate on a date that was announced months earlier.

Fifth, success that is not success. Several failures arrive as HTTP 200:

• A Responses call with status: "incomplete" and incomplete_details.reason: "max_output_tokens". Reasoning tokens bill as output and consume the same budget, so a reasoning model with a small cap can spend everything thinking and return nothing.
• A Chat Completions response with finish_reason: "content_filter" and empty or partial output.
• A streaming response that has already sent HTTP 200 and then delivers an error event mid-stream, with codes such as server_error and too_many_requests.
• A response where content_filter_results.error.code is content_filter_error, meaning the filtering system was unavailable and the request completed unfiltered.

A gateway or client library that only inspects the status code will count all of these as successful. Your platform's shared client wrapper should classify them explicitly and emit them as separate metrics.

Sixth, quota that is locked by a deletion. Deleting a resource soft-deletes it. If it is deleted through the API or CLI while deployments still exist, its quota stays reserved for 48 hours unless you purge it. A teardown script that deletes and recreates an environment can therefore fail the recreate for two days. Purge with az cognitiveservices account purge, which needs Contributor at subscription scope to view and purge deleted accounts. Confirm the flags against the current manage-resources page before you put it in automation.

Deployment types as a portfolio

At Mid level you picked a deployment type for one workload. As an owner you manage a portfolio in which each type answers a different question, and the main skill is keeping teams from defaulting to the wrong one.

The deployment types, by the SKU name you will see in ARM and Bicep, are GlobalStandard, DataZoneStandard, Standard (regional), GlobalProvisionedManaged, DataZoneProvisionedManaged, ProvisionedManaged (regional provisioned), GlobalBatch, DataZoneBatch, and DeveloperTier, which is only for evaluating fine-tuned models, lives 24 hours and has no SLA.

The data processing location is the property that decides which are allowed for a given workload. Global means processing in any Azure region. Data Zone means within the US, the EU (the EU Data Boundary, which can include EFTA countries) or APAC. Standard and Regional Provisioned mean within the Azure geography. Data at rest remains in the resource's geography for all of them. For an employer in the Gulf or Egypt this is a procurement and compliance conversation, not an engineering one: a Global deployment processes prompts outside any particular country, and you should get the approved position from your compliance team in writing, not infer it. Note also that new models roll out in a fixed order, Global Standard first, then Global Provisioned, then Data Zone, then regional Standard and Provisioned last, with no guaranteed date. A policy of "regional only" therefore means you are routinely on older models.

Three of these are newer and need care.

Flex processing is in preview. Set service_tier: "flex" for a 50% discount; requests rejected for capacity are not billed. Since 25 September 2026, a model that does not support Flex returns HTTP 400 invalid_request_error, with no automatic fallback to Standard. At the time of checking, the only supported model is gpt-5.6-sol on Global Standard. This is a trap for a platform default: if you set service_tier: "flex" in a shared wrapper and a team switches model, they get a hard failure rather than a price change. Flex also needs a long client timeout (around 900 seconds) and an explicit fallback to default on 429.

Priority processing is requested per request with service_tier: "priority", or set on the deployment through "properties": {"service_tier": "priority"}. The request values are auto (follow the deployment), default, priority and flex. The header x-ms-service-tier overrides the body. Two operational limits matter: it is available on Global Standard and Data Zone Standard (US only, not the EU data zone, not regional) for model versions from 2025-12-01, and a ramp of more than 50% TPM growth in under 15 minutes can downgrade requests to standard, visible as service_tier: "default" in the response. Priority shares quota with standard, and downgraded requests bill at standard. If you sell Priority internally as "faster", you must also monitor the downgrade rate, or the promise is empty.

Batch is asynchronous JSONL at 50% off with a 24-hour target, a separate enqueued-token quota, and completion_window fixed at "24h". Recommend dynamic quota for Global Batch. Two hazards for a platform: a CLI-created resource without a proper https://<name>.openai.azure.com custom subdomain does not work out of the box with Global Batch, and a UTF-8 BOM in the JSONL fails validation with a message about extracting a deployment name. Both of those belong in the onboarding checklist, not in each team's debugging session.

Quota, tiers and pooling

Quota is the mechanism by which Microsoft rations capacity to you, and your job is to ration it again to your teams. Two changes in 2025 and 2026 altered how that works, and many internal guides still describe the old world.

Quota tiers replaced the old Default and Enterprise levels. There is a free tier (Tier 0) and Tiers 1 to 6. Tiers upgrade automatically based on usage, Enterprise Agreement or Microsoft Customer Agreement Enterprise status and payment history, and a preview policy, tierUpgradePolicy: "NoAutoUpgrade", opts a subscription out. Your tier decides the default quota for each model, and for the newest models some tiers must request quota while higher tiers have it by default. Read your tier through the control plane, using a preview API version at the time of writing:

SUB=$(az account show --query id -o tsv)
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/quotaTiers?api-version=2025-10-01-preview"

Subscription-level pooling began after 7 May 2026, starting with the Realtime Translate and Realtime Whisper models and rolling out to all models. For Global Standard, one pool per model and version is shared across all regions. For Data Zone Standard, one pool per data zone. The consequence for a platform owner is large and easy to miss: the old practice of balancing a model's quota across regions to get more total capacity changes meaning when the pool is shared. Re-check, per model, whether quota is pooled in your subscription before designing a multi-region split around it.

Quota is a ceiling, not a guarantee. Having quota does not mean the capacity exists. This is most acute for provisioned deployments, where you can hold PTU quota and still fail to create a deployment because no capacity is available in that region. Standard types can return 429 for backend pressure even when you are well inside quota.

Two APIs let you manage this properly rather than by portal clicking. The usages API returns current consumption, with names such as OpenAI.Standard.gpt-4o in thousands of TPM:

LOC=eastus
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$LOC/usages?api-version=2024-10-01"

The Model Capacities API returns availableCapacity per location and SKU for a model and version, which is the only honest way to know whether a provisioned deployment can be created before you commit to it:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=gpt-5.1&modelVersion=2025-11-13"

Viewing quota requires the Cognitive Services Usages Reader role at subscription scope, which is a detail that routinely blocks a platform team's first dashboard. Quota changes in the portal take up to about 15 minutes, and extra quota is requested through the form at https://aka.ms/oai/stuquotarequest.

Allocating inside a subscription. Each deployment is assigned part of the quota, in capacity units where one unit is 1,000 TPM for standard deployments, and RPM is derived from it (older chat models 6 RPM per 1,000 TPM, newer models mostly 1 RPM per 1,000 TPM). The platform decision is how to carve the pool. Three schemes recur:

1. Shared deployment, gateway-enforced limits. One large deployment per model, with per-team limits enforced in API Management. Efficient, but a bug in the gateway becomes a quota incident.
2. Per-team deployments. Each team gets a deployment with its own allocation. Clean isolation of noisy neighbours, at the cost of stranded capacity: a team using 20% of its allocation holds the other 80% hostage.
3. Tiered. A shared deployment for development and low-priority work, dedicated deployments for the few production workloads that justify them.

Most platforms end at the third. Whatever you choose, record the allocation in code, not in the portal, because the portal is where unrecorded promises accumulate.

Provisioned throughput and spillover

A PTU (provisioned throughput unit) is reserved, model-independent capacity in a specific region. You pay an hourly rate per PTU whether or not you use it, or you buy a one-month or one-year Azure Reservation for a discount. The maximum is 100,000 PTU per deployment. What you buy is predictability: a stable latency profile and a capacity you do not share with other customers' bursts. What you give up is elasticity, because the meter runs at 3 a.m. exactly as it does at noon.

Sizing starts from measurement, not from a rule of thumb. The inputs to the PTU calculator (reachable from the Foundry portal at the path ai.azure.com/nextgen/goto/build/models/ptu-calculator) are requests per minute, average input and output tokens, the output-to-input ratio and the prompt cache rate. Take those from a real week of traffic on a Standard deployment, using ProcessedPromptTokens, GeneratedTokens and AzureOpenAIContextTokensCacheMatchRate. A PTU deployment sized from a forecast, rather than from observed traffic, is the single most common source of either a throttled production system or a six-figure idle bill.

The purchase order matters. A reservation does not guarantee capacity. The safe sequence is: query the Model Capacities API, create the deployment, confirm it succeeded and serves traffic, and only then buy the reservation against it. Buying first means paying for a discount on something you may not be able to create.

Utilization is the signal that a PTU deployment is full. When AzureOpenAIProvisionedManagedUtilizationV2 reaches 100%, new requests receive 429. Note the "V2" in the name: the V1 metric AzureOpenAIProvisionedManagedUtilization is deprecated, and dashboards copied from older blog posts still chart it.

Spillover is the feature that makes PTU usable as a platform default. A provisioned deployment can overflow to a standard deployment in the same resource when it returns 429, a long-context 400, 500 or 503. It requires a matching pay-as-you-go deployment, with the same model and version, in the same resource. There are two ways to enable it. For all traffic, set the deployment property spilloverDeploymentName on the provisioned deployment:

az rest --method put \
  --url "https://management.azure.com/subscriptions/$SUB/resourceGroups/rg-ai/providers/Microsoft.CognitiveServices/accounts/aoai-prod/deployments/chat-ptu?api-version=2024-10-01" \
  --body @ptu-deployment.json

{
  "sku": { "name": "GlobalProvisionedManaged", "capacity": 100 },
  "properties": {
    "model": { "format": "OpenAI", "name": "gpt-5.1", "version": "2025-11-13" },
    "spilloverDeploymentName": "chat-std"
  }
}

For one request, send the header x-ms-spillover-deployment: chat-std. If both are set, the deployment property wins. Responses tell you what happened through x-ms-spillover-from-deployment, x-ms-deployment-name and x-ms-spillover-error, and the metric dimension IsSpillover lets you chart it. The value of spillover is not just resilience. It lets you size PTU for the baseline and let the peak overflow to pay-as-you-go, which is usually cheaper than sizing for the peak. The failure mode is the opposite assumption: that spillover is free. Spilled traffic bills at standard rates and consumes standard quota, so a platform that sizes the standard deployment to zero because "PTU handles it" has built an outage that triggers exactly when load is highest.

One constraint deserves a sentence of its own. GPT-4.1 on provisioned deployments supports less than 128K tokens of context, so a long-context request returns HTTP 400. Spillover covers this case, which makes it a correctness feature for that model, not only a capacity one.

Security: the trust model

Security at this level is a set of decisions about what each identity can do, made once and enforced by the platform instead of being rediscovered by each team.

Keys are root. Azure OpenAI has two authentication methods. An API key sent in the api-key header grants full data-plane access with no role checks at all. Microsoft Entra ID, sent as Authorization: Bearer, is subject to RBAC and requires a custom subdomain on the resource. The consequence is blunt: as long as keys are enabled, your RBAC design can be bypassed by anyone who holds a key. The platform position is keyless. Set disableLocalAuth: true on the resource, authenticate with a managed identity wherever code runs in Azure, use workload identity federation or a certificate-backed service principal from CI outside Azure, and treat any remaining key as a documented exception stored in Key Vault, rotated between key1 and key2.

resource aoai 'Microsoft.CognitiveServices/accounts@2025-06-01' = {
  name: 'aoai-prod-gulf'
  location: 'swedencentral'
  kind: 'AIServices'
  sku: { name: 'S0' }
  identity: { type: 'SystemAssigned' }
  properties: {
    customSubDomainName: 'aoai-prod-gulf'
    disableLocalAuth: true
    publicNetworkAccess: 'Disabled'
    allowProjectManagement: true
  }
}

The region in that example is a placeholder. Choose it from the current deployment-type and model availability pages and your residency requirement.

Roles must be exactly as wide as the job. On an Azure OpenAI resource:

• Cognitive Services OpenAI User can run inference and use the playground. It cannot read keys or create deployments. This is the role for applications.
• Cognitive Services OpenAI Contributor adds deployments, fine-tuning, stored completions and On Your Data sources.
• Cognitive Services Contributor manages the control plane, keys and guardrails, but cannot make Entra inference calls. Owners and Contributors being unable to call the model surprises people every time.
• Cognitive Services Usages Reader reads quota, at subscription scope only.

On a Foundry resource the roles were renamed. The documentation advises using the role GUIDs in code while the rename rolls out, since the names are in flux:

Foundry User is the least-privilege developer role, Foundry Project Manager adds publishing agents and assigning Foundry User, Foundry Account Owner creates accounts and projects with no data actions, and Foundry Agent Consumer can call agent endpoints only, scoped even to a single agent. Do not use "Azure AI Developer" for Foundry projects, because it belongs to Azure Machine Learning workspaces and hubs. Fine-tuning needs both planes, so it needs Foundry Owner, or Foundry User plus Foundry Account Owner.

az role assignment create \
  --assignee-object-id "$APP_MI_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" \
  --scope "$(az cognitiveservices account show -n aoai-prod-gulf -g rg-ai --query id -o tsv)"

Role assignments take up to five minutes to propagate. Build that delay into provisioning pipelines, or the first deploy after granting access will fail with a 403 that vanishes on retry.

Wildcard roles are an upgrade hazard. After a Foundry upgrade, the role Cognitive Services User grants all Foundry features, whereas Cognitive Services OpenAI User stays OpenAI-only. Any broad role or Azure Policy that was acceptable on an OpenAI-only resource must be reviewed before the upgrade, not after.

What Microsoft commits to, and what you still own. The official data-privacy commitments are that prompts, completions, embeddings and training data are not available to other customers or to OpenAI and are not used to train foundation models, and that models are stateless. Stored data (files, Responses API state, stored completions, fine-tuned models) stays in the resource's geography and is AES-256 encrypted, with optional customer-managed keys. Separately from that, abuse monitoring may store flagged prompts and completions in the geography for human review. Customers can apply for modified abuse monitoring, and if approved the resource shows ContentLogging: false. Know which of these applies to each of your resources, because a regulated team will ask, and the honest answer comes from the resource JSON rather than from a slide.

You also own things the commitments do not cover. The Responses API is stateful by default, with a 30-day retention, so every call stores the conversation unless the caller sets store=false. A platform that promises "we keep nothing" must enforce that in the shared wrapper. And prompt injection is yours: guardrails offer Prompt Shields for user and indirect prompt attacks, but an agent with tools and a broad role is an authorization problem first.

Network isolation

Setting publicNetworkAccess to Disabled and adding a private endpoint is the standard enterprise posture, and it is also where platforms lose a week to DNS. The private endpoint must sit in the same region and subscription as the virtual network.

A Foundry resource exposes three FQDNs, and each needs private DNS: {custom-domain}.openai.azure.com for the OpenAI v1 API, {custom-domain}.services.ai.azure.com for Foundry APIs, projects and the /anthropic path, and {custom-domain}.cognitiveservices.azure.com. The matching private zones follow the standard private-link pattern, which is privatelink.openai.azure.com, privatelink.services.ai.azure.com and privatelink.cognitiveservices.azure.com. Confirm the zone names on the private-link page for your resource type, since the names are what you will encode in Terraform. An Azure OpenAI resource exposes only the /openai/v1 surface.

The failure signatures are specific and worth teaching your support team:

• A name that resolves to a public IP from inside the VNet. The private zone is missing, not linked to that VNet, or your custom DNS does not forward to 168.63.129.16. The call then fails because public access is disabled, and the error looks like a firewall problem.
• Private endpoint stuck in Pending. The approver lacks Contributor or Owner on the target resource.
• Works for openai.azure.com, fails for the project endpoint. After a Foundry upgrade, re-create the private endpoint so that the services.ai.azure.com and cognitiveservices.azure.com IP configurations exist.
• A VNet built on 172.17.0.0/16. Do not use that range.

The trusted-services bypass covers Microsoft.CognitiveServices, Microsoft.Search and Microsoft.MachineLearningServices, which is what lets Azure AI Search reach the resource. A private search indexer needs executionEnvironment: "Private". Private-network and customer-managed-key resources were not initially selected for the automatic Foundry upgrade, which is useful to know but not something to rely on, since selection criteria are Microsoft's to change.

Verification belongs in the pipeline, not in a person's memory:

# run from inside the VNet (a jump host, build agent or a container in the subnet)
nslookup aoai-prod-gulf.openai.azure.com
nslookup aoai-prod-gulf.services.ai.azure.com
nslookup aoai-prod-gulf.cognitiveservices.azure.com
# every answer must be a private address, not a public one

Multi-tenancy and the gateway

Once several teams share the platform, you need three things the service does not provide by itself: per-team limits that do not depend on the team's good behaviour, a way to attribute cost, and a single place to implement failover. The standard answer is a gateway, and on Azure that means API Management as an AI gateway.

The service-specific policies in current documentation use the llm-* names, and the older azure-openai-* names should be treated as unchecked:

• llm-token-limit enforces per-key tokens-per-minute or token quotas, and estimate-prompt-tokens lets it reject a request before it consumes backend quota.
• llm-emit-token-metric emits token counts with custom dimensions, which is the basis for chargeback.
• llm-semantic-cache-lookup and llm-semantic-cache-store implement a semantic cache, and need Azure Managed Redis or RediSearch plus an embeddings deployment.
• llm-content-safety applies content safety checks in the gateway.

Backends are configured as load-balancer pools with round-robin, weighted, priority or session-aware routing, and a circuit breaker that honours Retry-After. The priority pool is the one that matters most: PTU deployment first, standard second, a second region third. API Management authenticates to the backend with its managed identity, so callers hold only an APIM subscription key, which you can issue, rate-limit and revoke per team without touching the model resource.

<policies>
  <inbound>
    <base />
    <llm-token-limit
        counter-key="@(context.Subscription.Id)"
        tokens-per-minute="200000"
        estimate-prompt-tokens="true"
        remaining-tokens-header-name="x-remaining-tokens" />
    <llm-emit-token-metric namespace="llm">
      <dimension name="Subscription ID" />
      <dimension name="API ID" />
    </llm-emit-token-metric>
    <authentication-managed-identity resource="https://ai.azure.com" />
  </inbound>
</policies>

Treat the XML above as a shape to adapt, and validate attribute names against the current API Management policy reference before using them, because I have kept it to the policies the documentation names and not to every attribute.

Isolation levels, from cheapest to strongest. The right choice follows the failure you cannot tolerate.

1. A shared resource and deployment, with APIM subscription keys and token limits per team. Cheapest. A gateway misconfiguration or a shared-quota burst affects all.
2. Per-team deployments on a shared resource. Isolates TPM and guardrail configuration, not keys, network or data.
3. Per-team Foundry projects. Isolates RBAC and agent data. Role assignments can be scoped to a single agent.
4. Per-team resources. Isolates network, keys, encryption and region.
5. Per-team subscriptions. Isolates quota and billing. Quota is per subscription, so this is also the only way to give a team a quota that nobody else can consume, and with pooling this matters more than before.

For attribution, the platform emits a team identifier on every request, in a gateway dimension or in a billing tag. Custom headers are a tempting transport, but at most ten pass through, more than ten returns HTTP 431, and Microsoft has said future versions will not pass custom headers through, so do not design a product feature around them.

Cost: attribution, premiums and the meters you cannot stop

Standard deployments bill per input token, cached input token and output token, by model. Reasoning tokens bill as output tokens, so a reasoning_effort change is a cost change. GPT-5.6 and GPT-6 models have separate short-context and long-context price categories, decided by the number of input tokens in the request. A team that starts sending a 400K-token context to save on retrieval work has changed its price category without changing a line of billing code, and a request-level token count in your telemetry is the only way you will notice.

The adjustments are these: Batch is 50% off Global Standard; Flex is 50% off Standard with unbilled capacity rejections; Priority is a premium rate, with downgraded requests billed at standard; provisioned is hourly per PTU regardless of usage, or a reservation discount, with spillover traffic billed at standard. Fine-tuning bills training hours plus hosting, global training is cheaper but has no data residency, and the developer tier is the cheapest way to evaluate a fine-tune. Upgrading to Foundry does not raise Azure OpenAI prices. Check current rates on the pricing page, never in a slide deck.

The cost levers you can actually pull, in order of typical value:

1. Move delay-tolerant work to Batch or Flex.
2. Right-size max_output_tokens and reasoning effort.
3. Raise the prompt cache hit rate by putting stable content at the start of prompts. You can see it in AzureOpenAIContextTokensCacheMatchRate.
4. Move a stable baseline to provisioned capacity.
5. Route simple requests to cheaper models, by explicit design or with the model router (GA version model-router 2025-11-18), which selects an underlying model per prompt. Evaluate it on your own traffic, because routing changes quality as well as price.

Tracking. In Cost Analysis, filter by the billing tag deployment, and note that Flex has its own meters. The metric FoundryModelEstimatedCost is in preview and is an estimate, not an invoice. Quota is not a budget: the legacy approach of capping spend by capping TPM is explicitly not best practice, because it converts a cost control into an availability incident. Budgets and alerts belong in Azure Cost Management, with the throttle as a last resort.

For chargeback, join gateway token metrics (by team) to the per-model price. Reconcile monthly against the invoice and publish the reconciliation error, because a chargeback model finance does not trust is worse than none.

Observability that finds problems

Platform metrics arrive automatically at one-minute granularity. The ones to build on, under their REST names, are AzureOpenAIRequests (dimensions include StatusCode, ModelDeploymentName, ModelName, ModelVersion, StreamType, IsSpillover, ServiceTierRequest, ServiceTierResponse, ApiName, OperationName and Region), ProcessedPromptTokens, GeneratedTokens, TokenTransaction, ActiveTokens for PTU, AzureOpenAIContextTokensCacheMatchRate, AzureOpenAIProvisionedManagedUtilizationV2, AzureOpenAIAvailabilityRate, the guardrail metrics RAIRejectedRequests, RAIHarmfulRequests and RAITotalRequests, and a family of latency metrics: AzureOpenAITimeToResponse, AzureOpenAITTLTInMS (time to last byte), AzureOpenAINormalizedTTFTInMS, AzureOpenAINormalizedTBTInMS and AzureOpenAITokenPerSecond.

Two warnings from the documentation shape the dashboards. Do not use the legacy Cognitive Services metrics (Latency, TotalCalls, ClientErrors, SuccessRate), because the docs say they mislead for Azure OpenAI. And the tokens-per-second, time-to-response and time-between-tokens metrics are not available for Standard deployments, so your latency view for pay-as-you-go traffic comes from the client or the gateway, not from the platform. Always chart latency next to token counts, since a latency rise with longer outputs is not an incident.

Resource logs need a diagnostic setting. Categories are Audit, RequestResponse, Trace, AzureOpenAIRequestUsage and ManagedNetworkEvent, and they land in the AzureDiagnostics table, with metrics in AzureMetrics. Crucially, prompt and completion bodies are not in these logs. A security team asking "can we see what users sent?" will assume the diagnostic logs answer it. They do not. Logging bodies means gateway LLM logging or application-side tracing, and that is a data-handling decision with its own retention and access rules, which you should make explicitly. For application tracing, see Langfuse and OpenTelemetry.

AzureDiagnostics
| where Category == "RequestResponse"
| summarize requests = count(), p95_ms = percentile(DurationMs, 95) by bin(TimeGenerated, 5m), ResultSignature
| order by TimeGenerated desc

Treat that as a starting query and check the columns present in your workspace, since the table is schema-on-read and column names depend on the category. Alerts are Azure Monitor metric and log alerts, and Azure Monitor Baseline Alerts offers a starting set.

The set of alerts a platform needs is small:

1. 429 ratio per deployment, above a threshold for several minutes.
2. PTU utilization above 90% sustained, and spillover rate above a threshold.
3. Availability rate below target.
4. A guardrail rejection spike, which is either an attack or a broken prompt change.
5. Days to retirement for every deployed model, from the lifecycle API, alerting well in advance.
6. A new deployment or role assignment outside the pipeline, from the activity log.

Lifecycle, upgrades and migrations

Models are the part of this platform that expires on a schedule. The current policy is that a GA model is deprecated to new customers at 12 months and retires at 18 months from launch; retired models return 410 Gone. GA retirement notice comes at least 60 days ahead, preview models get at least 30 days and are force-upgraded, and Anthropic, DeepSeek, Fireworks and Mistral GA models follow a 12-month lifecycle. Retirement dates cannot be extended. At the time of checking, the schedule included the o-series on 2026-11-19, gpt-4.1-nano on 2026-10-14 and gpt-4o 2024-05-13 on 2026-12-09. Those dates will be history by the time you read this, so read the live schedule rather than this paragraph.

Query the lifecycle from the control plane and build the alert from it:

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$LOC/models?api-version=2024-10-01" \
  --query "value[?model.name=='gpt-4.1-mini'].model.{v:version,status:lifecycleStatus,dep:deprecation.inference}"

A migration process that works has these steps. Subscribe a shared mailbox to Azure Service Health advisories for the service name "Azure OpenAI Service", since there is no separate Foundry entry and emails otherwise go only to subscription owners. Inventory every deployment and its callers. Evaluate candidate replacements on your own data as soon as they exist, because Microsoft names a replacement only about 90 to 120 days ahead and replacements reach Global Standard about 90 days before retirement (provisioned regions about 30 days before). Then roll out by traffic percentage behind the gateway. For evaluation harnesses see Promptfoo and RAGAS.

Behaviour changes hide in upgrades. gpt-5.1 defaults to reasoning_effort: none, so code moved from older reasoning models must pass an effort explicitly. GPT-5.6 on Chat Completions rejects function tools combined with reasoning, even when the caller never sent reasoning_effort, because the default is medium; the fix is the Responses API or reasoning_effort: "none". The o-series parameter rules (max_completion_tokens, no temperature) differ again from GPT-6. Pin your SDK versions and test upgrades of openai and azure-ai-projects. The Assistants API retired on 2026-08-26, and classic Foundry Agent Service retires 2027-03-31, so an inventory of code calling /assistants is a real deliverable. Embedding models cannot be upgraded in place, because changing one forces a full re-embed of the corpus.

The Foundry upgrade deserves its own line in the plan. Microsoft is auto-upgrading eligible Azure OpenAI resources to Foundry. Scheduled resources show a foundryAutoUpgrade block at ARM api-version 2026-01-15-preview and a "Resource upgrade" blade in the portal. To opt out, set foundryAutoUpgrade: { mode: 'Disabled' } in Bicep or AzAPI, or use "Defer upgrade" in the portal. In Terraform, azurerm 4.56.0 or later is required to change kind from OpenAI to AIServices without destroying the resource. Review wildcard roles and policies first, as above. Rollback fails if the region exceeds 30 Azure OpenAI resources, and the upgrade fails if it exceeds 100 AIServices resources.

Governance and infrastructure as code

Everything above decays if it lives in the portal. Declare resources, deployments, role assignments and private endpoints in code, with Bicep (Microsoft.CognitiveServices/accounts@2025-06-01 and the child deployments type) or Terraform (azurerm_cognitive_account and azurerm_cognitive_deployment, with AzAPI for preview properties). See Terraform. A deployment in code carries its sku, capacity, versionUpgradeOption, raiPolicyName and, for PTU, spilloverDeploymentName. Reviewing a pull request that changes any of those is the cheapest control you have.

Azure Policy is the enforcement layer. You can restrict deployment SKUs with the alias Microsoft.CognitiveServices/accounts/deployments/sku.name, which stops a team from creating a regional provisioned deployment you never approved, and there is a built-in policy for allowed models. Policies that require disableLocalAuth and disabled public access make the secure posture the only available one.

Guardrails are configured per deployment as RAI policies. The defaults block medium and high severity. Annotate-only or no filtering on completions requires approval for modified content filtering, and that approval should be tracked as an exception with an owner. Who may weaken a guardrail is an RBAC question, so give that permission to very few identities and alert on policy changes. Guardrails are a layer of defence, not a certification, and the guardrails metrics tell you when they fire. Pair them with your own checks, for which Guardrails covers application-level validation.

Incident playbooks

Three playbooks cover most pages.

A spike of 429s. First read the text, to separate the three kinds described earlier. If it is your TPM, check max_output_tokens and burst shape before raising quota, because the estimate includes the cap. If it is backend pressure, route to another deployment or region at the gateway; no quota edit helps. If the limit header is below your TPM, back off and spread load. Then check whether PTU spillover is configured and whether IsSpillover is rising.

A model stops working. 410 Gone means retirement. Check the lifecycle API, point the deployment at the replacement version if versionUpgradeOption allows, or create a new deployment and shift traffic at the gateway. A 404 DeploymentNotFound is usually a wrong or case-mismatched deployment name, a wrong endpoint, or a deployment created in the last few minutes.

Authentication fails after a change. 401 with an Entra token means no custom subdomain or the regional endpoint was used. 403 means a missing data-plane role, since Owner and Contributor are not enough, or a role assignment still propagating. "No permissions to create agents" for an Owner means they need a Foundry role. A call that works locally but not in Azure usually means the managed identity is disabled, the role went to the user instead of the identity, or a user-assigned identity was not named through AZURE_CLIENT_ID.

Each playbook ends the same way: record what was learned in the runbook and, if a human noticed before an alert did, add the alert.

Where Azure OpenAI stops

Azure OpenAI is a strong choice when you need OpenAI-family models under Azure's identity, network, compliance and billing, but it is not the right answer to everything.

• You need many providers behind one interface. The Foundry resource hosts other providers' models and an /anthropic path, but a vendor-neutral proxy such as LiteLLM is a better abstraction when the portfolio spans clouds. Compare AWS Bedrock if your estate is on AWS.
• You need the newest features first. Features reach the OpenAI API before Azure, and regional Standard types receive models last. The direct OpenAI API may be right for experiments, provided your data policy allows it.
• You need to host open weights on your own hardware. Latency, cost and data control can favour self-hosting, which is outside what a managed deployment gives you.
• Your machine-learning platform is the centre of gravity. Training and serving classic models belongs in Azure Machine Learning.
• Your residency requirement is stricter than any available deployment type. Then no configuration on this page fixes it.

Running it as a platform for other teams

The service is the easy part. The product you operate is the paved road: a request form or pull request that yields a compliant resource, a shared client wrapper, a gateway endpoint and a runbook.

What the paved road contains: an IaC module that creates a keyless, private resource with diagnostics wired; a client wrapper with one retry layer, store=false as the default where that matters, finish_reason and incomplete-status classification, and a team tag on every call; a published model catalogue with approved models and retirement dates; a quota process with a named owner, which also means saying no to requests; and a support rotation that can read the failure signatures in this guide. Publish a service-level objective in terms the service can actually deliver, such as gateway-measured availability and added latency, not model quality.

Communicate change in advance and in writing. Retirements, the Foundry upgrade, a quota-tier change and a default-parameter change each deserve a dated notice with an owner and a rollback plan. Teams trust a platform that says what changes and when.

The review checklist

Use this for design documents and pull requests.

1. Is every identity keyless, with disableLocalAuth: true and the narrowest role, applied at resource or project scope?
2. Is the data processing location approved by compliance in writing?
3. Are all three private DNS names verified from inside the network?
4. Is there exactly one retry layer, and does it honour retry-after-ms?
5. Are max_output_tokens and reasoning effort set deliberately?
6. Does the client classify HTTP 200 failures: incomplete, content_filter, mid-stream errors?
7. Is quota allocation recorded in code, and is each PTU deployment backed by a measured size, spillover and an owner?
8. Is every model's retirement date on a calendar with an alert?
9. Are foundryAutoUpgrade, versionUpgradeOption and the Terraform provider version decided explicitly?
10. Is there a second region or deployment, and has failover been tested?
11. Are cost attribution and a monthly reconciliation in place?
12. Is the choice of Azure OpenAI over the alternatives written down?

The complete picture

Quota is rationed twice, once by Microsoft and once by you. Identity is keyless and the roles are narrow. Capacity is a portfolio. Models expire on a schedule you track. Observability watches the failures the status code hides. And the whole arrangement is code, reviewed like code.

Where the series leaves you

You started by sending a first request to a deployment. You can now own the platform behind thousands of them: design the topology, ration the quota, secure the identities, isolate the network, attribute the cost, survive a retirement and a regional event, and decide when to use something else. Natural next reads are LiteLLM for a provider-neutral gateway, Langfuse for application-level traces, and Terraform for the code that holds it together.

Sources

• Azure OpenAI overview
• API version lifecycle and the v1 API
• Deployment types
• Quotas and limits
• Manage quota and rate limits
• Provisioned throughput
• Spillover traffic management
• Priority processing
• Flex processing
• Batch
• Model retirement schedule
• Model lifecycle and support policy
• RBAC for Foundry
• Upgrade Azure OpenAI to Foundry
• Automatic upgrades to Foundry
• Network isolation with private link
• Data, privacy and security
• Monitoring data reference
• API Management AI gateway capabilities
• Pricing

Part three of three. This is a cumulative review for someone who owns Azure OpenAI as a platform: system-design prompts built around the service, incident questions, and the conversations where you argue a trade-off to people who do not care about tokens. Read each prompt, answer it aloud before reading the model answer, and compare what you covered with what a strong answer covers. About fifty minutes.

The thirty-second answer

Azure OpenAI runs OpenAI-family models under Azure's identity, networking, compliance and billing. You create a resource, deploy a model to it under a name you choose, and call that deployment name in the model field. The platform job is to ration quota between teams, keep identities keyless, isolate the network, survive model retirements and attribute cost.

If you can say that cleanly, then add the naming point unprompted: the product is now Azure OpenAI in Microsoft Foundry Models, content filters are called guardrails, and the data-plane API is the v1 API that replaced monthly api-version strings.

System-design prompts

Design a shared Azure OpenAI platform for forty product teams

What the interviewer wants: a topology, an isolation argument and a way to say no.

A strong answer starts with the requirement that drives everything, which is the isolation each team needs, and only then draws boxes. Quota is per subscription, so teams in one subscription compete for one pool. A resource isolates network, keys, region and encryption. A project isolates RBAC and agent data. A deployment isolates only TPM allocation, guardrail and SKU. So the design is layered: a gateway (API Management as an AI gateway) in front of a small number of resources, with each team issued an API Management subscription key, not a model key. The gateway enforces per-team token limits with llm-token-limit, emits per-team token metrics with llm-emit-token-metric for chargeback, authenticates to the backend with its managed identity, and holds the failover logic in a backend pool with a circuit breaker.

The resources are keyless (disableLocalAuth: true), reached through private endpoints, with diagnostic settings to Log Analytics. Deployments are declared in Bicep or Terraform and recorded in a model catalogue that includes retirement dates. Teams with strict isolation needs, for example a regulated workload, get their own resource or subscription. The quota process has an owner and a documented "no", because quota is the product.

What weak answers miss: they put all teams on one deployment with no per-team limits, so one burst becomes everyone's outage. Or they give every team a resource and have no idea how to monitor thirty-eight of them. Or they never mention how a team's identity is separated from the model credential.

Follow-up to expect: "What if one team needs guaranteed latency?" Answer: a dedicated provisioned deployment for that workload, sized from observed traffic, with spillover to a standard deployment, behind the same gateway so it is invisible to other teams.

Design for a latency-sensitive customer-facing assistant that must stay up

What the interviewer wants: a capacity story and a failure story.

Start with the capacity question. Pay-as-you-go Standard types share backend capacity, and a 429 for backend pressure is not fixed by more quota. For steady volume and a latency commitment you use provisioned throughput (PTU). You size it from measurement, taking requests per minute, average input and output tokens, the output-to-input ratio and the prompt cache rate from real traffic and putting them in the PTU calculator. You check capacity with the Model Capacities API, create the deployment, and only then buy the reservation, because a reservation does not guarantee capacity.

Then the failure story. PTU is sized for the baseline and the peak spills over to a standard deployment of the same model and version in the same resource, via spilloverDeploymentName. Beyond one deployment, a second resource in another region sits behind the gateway in a priority pool, with a circuit breaker that honours Retry-After. Client behaviour completes the picture: one retry layer, backoff that respects retry-after-ms, and a timeout budget.

What a strong answer adds: monitoring of AzureOpenAIProvisionedManagedUtilizationV2 (the V1 metric is deprecated), the IsSpillover dimension, and a note that spillover traffic bills at standard rates and uses standard quota, so the standard deployment cannot be sized to zero. Also the 200-status failures: finish_reason: content_filter, incomplete status and mid-stream errors.

What weak answers miss: treating PTU as a guarantee of uptime. It is a guarantee about a capacity profile, and a region-wide event still takes it down without a second region.

Design an offline enrichment pipeline over 40 million documents

What the interviewer wants: you to pick the cheap path and explain its limits.

The answer is Batch, because the work tolerates delay. A Global Batch deployment gives 50% off, a 24-hour target and a separate enqueued-token quota. Input is JSONL with the same deployment name on every line, up to 100,000 requests and 200 MB per file, so 40 million documents means a sharding job and a scheduler that tracks many batches. completion_window must be "24h", the job is polled at most every 60 seconds, and a successful job still carries an empty error file. Enable dynamic quota for Global Batch, and check the first failures: a UTF-8 BOM fails validation with a message about a missing deployment name, and a CLI-created resource without a custom subdomain does not work out of the box with Global Batch.

The senior points are what surrounds it. Idempotency by custom_id. Re-running only failed lines. Cost reconciliation, since long context has its own price category and reasoning tokens bill as output. And a data-residency check: Global Batch processes anywhere in Azure, so a regulated corpus may require Data Zone Batch.

Follow-up: "Could Flex do this?" Flex is in preview, gives the same 50% discount for delay-tolerant online calls, but at the time of writing supports only one model and returns HTTP 400 for unsupported ones, so it is not a safe default.

Design private, keyless access from AKS and from on-premises

What the interviewer wants: identity and DNS, not just "use a private endpoint".

Identity first. Workloads in Kubernetes use workload identity federation or a managed identity; on-premises or other-cloud callers use a service principal with a certificate or federated credential. Roles go at the narrowest scope: Cognitive Services OpenAI User for inference on an Azure OpenAI resource, Foundry User on a Foundry project. disableLocalAuth: true removes keys, which matters because a key bypasses RBAC completely. The token audience for the data plane is https://ai.azure.com/.default.

Network second. publicNetworkAccess: Disabled, a private endpoint in the same region and subscription as the VNet, and private DNS for all three FQDNs of a Foundry resource. For on-premises, DNS conditional forwarding must reach a resolver that can answer the private zones, and custom DNS in Azure must forward to 168.63.129.16. The classic failure is a name that resolves to a public IP from inside the network, which looks like a firewall problem. For a Foundry upgrade, re-create the private endpoint so the extra FQDNs have IP configurations.

What strong answers verify: a pipeline step that resolves all three names from inside the network, not only the one the application uses.

Incident-response questions

"429s started an hour ago. Walk me through it."

Read the message first, because there are three different problems. "Rate limit is exceeded" is your deployment's TPM or RPM. "System is experiencing high demand" is backend capacity. A 429 whose x-ratelimit-limit-tokens is lower than your TPM is a temporary adjustment on the shared pool. For the first kind, remember admission uses an estimate based on prompt characters plus max_tokens, so check whether someone raised the output cap, and RPM is enforced in one- or ten-second windows, so look for a burst invisible at one-minute resolution. Check retry multiplication, where an SDK with max_retries and an application library both retry. For the second and third kinds, route elsewhere at the gateway and back off. Only then ask for quota.

A strong answer ends with prevention: a ratio alert on 429 per deployment, and one retry layer.

"A model that worked yesterday returns 410."

That is a retirement. 410 Gone is the documented response for a retired model version. You check the control-plane models API for lifecycle status, remembering the trap that the API's Deprecating means "Deprecated" in the documentation and the API's Deprecated means retired. If the deployment had versionUpgradeOption at NoAutoUpgrade, it stops at retirement by design. The fix is a new deployment of the replacement with traffic shifted at the gateway. The postmortem question is why there was no alert, since retirement dates are announced at least 60 days ahead for GA models and cannot be extended.

Follow-up: "How would you have known earlier?" Azure Service Health advisories to a shared mailbox, and a scheduled job that reads lifecycle data and alerts at a fixed number of days.

"After the upgrade to Foundry, the private-endpoint calls to agents fail but inference works."

Inference uses the openai.azure.com name, which the original private endpoint covered. Agents and projects use services.ai.azure.com, and the third name is cognitiveservices.azure.com. After the upgrade the endpoint needs re-creating so all three IP configurations exist, and the private DNS zones need records for all three. Verify by resolving each name from inside the network. A strong answer also mentions checking roles: "Cognitive Services User" grants all Foundry features, whereas "Cognitive Services OpenAI User" stays OpenAI-only, so an agent permission failure may be a role gap rather than DNS.

"Our bill doubled with no traffic change."

Candidates: reasoning effort changed or a model default did, so reasoning tokens (billed as output) went up; a prompt crossed into a long-context price category, decided by input tokens; the cache hit rate dropped because a prompt change moved stable content later; a Priority deployment property started billing the premium rate; a provisioned deployment was created and left idle; or work moved off Batch. The method is to break cost by deployment and token type using Cost Analysis with the deployment billing tag, then compare token metrics by model and ServiceTierResponse. A good answer states that FoundryModelEstimatedCost is a preview estimate, not an invoice.

"A prompt-injection incident exposed other users' data through an agent."

This is an authorization failure before it is a model failure. The agent held a broad role or tool access, and the model was asked to enforce a boundary the system should have enforced. Guardrails and Prompt Shields reduce risk, and indirect-attack detection benefits from delimiting untrusted documents, but the structural fix is least privilege for the agent identity (Foundry Agent Consumer can be scoped to one agent), tool permissions checked server-side, and per-user data filters applied outside the model. Also check data handling: Responses API state is stored for 30 days unless store=false, and diagnostic logs do not contain prompt bodies, so the investigation depends on what tracing you enabled in advance.

Trade-off questions

PTU or Standard?

Standard bills per token and has no commitment, but shares backend capacity and can return 429 for pressure. PTU bills hourly regardless of use (or at a reservation discount), gives a stable capacity profile, and needs sizing. The deciding facts are the shape of traffic and the cost of a slow response. Steady high volume with a latency requirement favours PTU for the baseline, with Standard absorbing the peak through spillover. Spiky or low-volume traffic favours Standard, with Priority processing as a middle option for latency without commitment.

Global, Data Zone or Regional?

Global gives the newest models, the broadest availability and the highest default quota, and processes in any Azure region. Data Zone keeps processing inside the US, the EU or APAC. Regional keeps it within the Azure geography but receives new models last. The question that decides it is a compliance one, and the engineer's job is to turn "regional is safer" into a precise statement: what data, which law, which processing location. At rest, data stays in the resource's geography in all cases.

One resource per team, or a shared gateway?

A shared gateway centralises limits, metrics and failover with the least overhead, but concentrates risk. Per-team resources isolate network and keys, but multiply what you monitor, patch and upgrade. Per-team subscriptions are the only way to isolate quota. Choose the weakest isolation that satisfies the requirement and document the trigger for stepping up.

Azure OpenAI or the OpenAI API directly?

Azure wins on identity integration, private networking, Azure billing and compliance posture. The direct API tends to receive features first and can be simpler for experimentation. For a company already on Azure with data-handling rules, Azure is usually right; it still leaves open a gateway such as LiteLLM when many providers are in play.

Arguing a trade-off to leadership

Leadership questions are about money, risk and dates. Lead with the decision and the number, then the risk, then what you need.

"Why are we paying for capacity we are not using?" Provisioned deployments bill hourly whether used or not. Say what it buys: a latency commitment on a named product, quantified. Then show utilization, and offer the alternative with its risk: moving the baseline to Standard saves a defined amount, at the cost of 429 exposure on peak. A strong answer proposes spillover to reduce the PTU size and shows the saving.

"Can we just ask for more quota?" Quota is a ceiling, not capacity. More TPM does not cure backend pressure, and requests are rationed through a form, with tiers rising on usage and payment history. Explain what you will do instead: reduce wasted reservation by lowering output caps, move delay-tolerant work to Batch, and add provisioned capacity where the business case exists.

"A model is being retired. Why does this need a project?" Because retirement dates are fixed and cannot be extended, replacements behave differently (default reasoning effort, tool-calling rules, parameter support), and the cost of finding that out in production is an outage. Offer a plan: inventory, evaluate on our data, canary at the gateway, cut over, with owners and dates.

"Why not let each team pick its own AI vendor?" Argue from governance, not preference. A single entry point gives one place for identity, logging, limits, cost attribution and incident response. It does not forbid other vendors, but it makes the exceptions visible. This is where a provider-neutral layer can be the honest compromise.

"Is it safe?" Never answer with a slogan. State the commitments (prompts and completions are not available to other customers or to OpenAI and are not used to train foundation models; stored data stays in the resource's geography), then name the residual risks you own: abuse monitoring that may store flagged content for review unless modified monitoring is approved, stateful Responses storage, injection through tools, and key leakage. Say what controls address each.

Common questions with model answers

What is the difference between a model and a deployment? The model is the catalogue item and version. The deployment is a named instance with a SKU and capacity, and the model field takes the deployment name.

Why does Entra authentication fail with 401? The resource has no custom subdomain, or the regional endpoint was used instead of https://<name>.openai.azure.com.

Why does an Owner get 403 on inference? Owner and Contributor are control-plane roles. Inference needs a data-plane role such as Cognitive Services OpenAI User or, on Foundry, Foundry User.

What does the Responses API change? It is the recommended API, stateful with a 30-day default retention, supports chaining, tools, background mode and compaction, and uses max_output_tokens. The Assistants API retired on 2026-08-26, so assistants become agents or prompts, threads become conversations and runs become responses.

Which parameters do reasoning models reject? max_tokens, temperature, top_p, penalties and logprobs on the o-series and older GPT-5; GPT-6 models support logprobs, temperature and top_p. Use max_completion_tokens on Chat Completions and max_output_tokens on Responses.

What are the three token budgets people forget? The reasoning tokens inside the output cap, the estimated tokens used for rate limiting, and the long-context price category decided by input size.

How do you stop a deployment drifting from the declared state? Deployments in code, Azure Policy on SKU and model, and an alert on changes made outside the pipeline.

Sixty-second self-test

Answer each in one sentence, aloud.

1. Why does a deployment name, not a model name, go in the model field?
2. Name the three kinds of 429 and the right response to each.
3. Why is a key worse than a role?
4. What is the purchase order for PTU, and why?
5. What does spillover require, and what does it bill at?
6. Which unit isolates network and keys, and which isolates quota?
7. What do diagnostic logs not contain?
8. What does the API's Deprecated status mean?
9. Why can a Global Batch deployment fail on a CLI-created resource?
10. What must you check before a Foundry upgrade?
11. Which retry layer should exist, and how many?
12. Name three failures that arrive as HTTP 200.

If you could answer ten without hesitation, you can carry the rest of the interview on judgment. Related reading: LiteLLM, Langfuse and Terraform.

Part three of three. At this level the problems are operational: a quota pool shared by forty teams, a model retired on a date nobody tracked, a private endpoint that works for one of three names. Start with the error table, then the habits, then the review checklists and recipes you can paste into a pull request template.

Common errors at this level

Cumulative: everything from Beginner and Mid still applies. These cause incidents rather than failed first calls, and most only appear once real volume or many teams are involved.

Traps that only appear at scale

The output cap you never set

By default, a request may reserve a very large output budget against your quota. At low volume that is invisible. At volume, a thousand requests that each reserve far more than they use throttle a deployment that is mostly idle. The cure is a platform-level default for max_output_tokens, enforced in the shared client wrapper, with an exception process for the few callers that need more. Set it per use case from measured p99 output, plus a margin.

One shared deployment

A single large deployment looks efficient. It stays efficient until the first team with a batch job fills it, after which every interactive product is throttled. Isolate by workload class: interactive, background and bulk should not share a deployment. Bulk belongs on Batch, which has its own enqueued-token quota and takes the load off the interactive pool entirely.

Quota thinking that predates pooling

Many internal documents say "spread a model's quota over regions to get more". Subscription-level pooling began after 7 May 2026 and is rolling out to all models: Global Standard shares one pool per model and version across regions, and Data Zone Standard one pool per data zone. Before you design a regional split, read the current quota page for the model in question. A design built on a false belief about the pool will be discovered at the worst moment.

Spillover that spills into nothing

Spillover requires a pay-as-you-go deployment of the same model and version in the same resource. Teams create the provisioned deployment, set spilloverDeploymentName, and then allocate zero quota to the standard one because "PTU covers it". The overflow then fails exactly when load peaks. Spilled traffic also bills at standard rates, so the budget must expect it. Chart IsSpillover and alert when the share grows, since a rising share means the PTU is undersized for the baseline.

Reservations bought before capacity exists

A reservation does not guarantee capacity. Teams buy the discount and then fail to create the deployment. Check the Model Capacities API, create and test the deployment, and buy last. Keep a named owner on every provisioned deployment, because the hourly meter runs without any traffic.

Defaults that moved under you

gpt-5.1 defaults to reasoning_effort: none. GPT-5.6 on Chat Completions rejects tools with reasoning at the default effort. Reasoning models reject temperature and max_tokens, except that GPT-6 models accept logprobs, temperature and top_p. Each of these arrives as a change of model, not a change of code, so the failures are blamed on the wrong team. The mitigation is an evaluation suite and a contract test suite run against the candidate model before any cutover. See Promptfoo and RAGAS.

The Foundry upgrade nobody scheduled

Microsoft is upgrading eligible Azure OpenAI resources to Foundry. Teams discover it in the portal, after which a wildcard role means more than it did. "Cognitive Services User" grants all Foundry features, and "Cognitive Services OpenAI User" stays OpenAI-only. Inspect ARM JSON for foundryAutoUpgrade at api-version 2026-01-15-preview, decide per resource whether to upgrade or set foundryAutoUpgrade: { mode: 'Disabled' }, and review roles and policies first. In Terraform use azurerm 4.56.0 or later, or a kind change will plan a destroy.

The status words that mean the opposite

The lifecycle API's Deprecating means Deprecated in the documentation (still serving, closed to new customers) and its Deprecated means Retired. An alert keyed to the literal word "Deprecated" fires after the outage.

Trusting a gateway for what it does not log

An AI gateway is the right place to record who called what. It does not make logging free of consequences: bodies logged at the gateway are prompt data, with retention and access obligations. Decide deliberately, and keep the default at metadata only.

Platform-ownership habits

These are the practices that pay off most, in order of leverage.

1. Make keyless the only way. Enable disableLocalAuth: true through Azure Policy so that new resources cannot accept keys. This one control removes a class of leaks and makes every access attributable.
2. Put retirement dates in a system that pages. Read lifecycle data from the control plane on a schedule and alert at fixed horizons well before each date. Subscribe a shared mailbox to Azure Service Health for the service name "Azure OpenAI Service".
3. Own one client wrapper. A thin library with one retry layer, a default output cap, store=false where required, a team tag, and classification of 200-status failures removes half the incidents in this guide. Teams adopt it when it is easier than writing their own.
4. Record allocation in code. Quota assignments, deployment SKUs, upgrade options and guardrail names live in Bicep or Terraform (Terraform) and change through reviewed pull requests. The portal is a viewer.
5. Evaluate replacements early on your own data. Microsoft names a replacement only about 90 to 120 days before retirement and it reaches Global Standard about 90 days before. Do not wait for the announcement to begin evaluating candidates.
6. Test failover on a schedule. Disable a backend in a controlled window and observe. An untested secondary region is an assumption with a cost.
7. Reconcile cost monthly. Join gateway token metrics to price, compare to the invoice, and publish the gap. Use the deployment billing tag in Cost Analysis, and remember Flex has its own meters.
8. Write down the isolation decision. For each workload, the level (gateway key, deployment, project, resource, subscription) and the requirement that justified it. Review when requirements change.
9. Publish a change calendar. Retirements, upgrade windows, quota tier changes and default changes, each with an owner. Teams plan around what they can see.
10. Own the exceptions list. Keys still enabled, modified content filtering, abuse-monitoring status, public network access. Each has an owner, a reason and an expiry.

Review checklist for a pull request

Use this on any change to infrastructure code, gateway policy or the shared client.

• Identity. Does the change keep disableLocalAuth: true? Is every role at the narrowest scope, and are Foundry roles referenced by GUID while the rename rolls out?
• Network. Is publicNetworkAccess disabled where required, and do tests resolve all three FQDNs privately?
• Deployment. Are sku.name, capacity, model version and versionUpgradeOption explicit? Does a provisioned deployment declare spilloverDeploymentName with a matching standard deployment of the same model and version, and non-zero quota?
• Guardrails. Is the raiPolicyName the approved one? Does any change weaken severity thresholds, and is there an approval?
• Client. Is there exactly one retry layer? Is max_output_tokens set? Is store handled deliberately? Are incomplete status, content_filter and mid-stream errors classified?
• Parameters. Do the request parameters suit the target model: max_completion_tokens or max_output_tokens for reasoning models, no temperature where unsupported, explicit reasoning_effort for gpt-5.1 and later?
• Service tier. Is service_tier set to Flex or Priority only where the model and deployment type support it, with an explicit fallback?
• Gateway. Do llm-token-limit and llm-emit-token-metric carry the right counter key and dimensions? Does the backend pool have a circuit breaker honouring Retry-After?
• Observability. Does the change keep the diagnostic setting and the alerts? Are any dashboards using legacy Cognitive Services metrics or the V1 utilization metric?
• Lifecycle. Is the model's retirement date recorded, and is there an alert?
• Provider version. If Terraform, is azurerm at 4.56.0 or later where the kind could change?

Review checklist for a design document

Ask these of any proposal that adds a workload to the platform.

1. What is the workload, and what does failure cost? A design with no failure cost cannot justify a deployment type.
2. Which data enters the model, and where may it be processed? Name the processing location (Global, Data Zone, Regional) and the written compliance position. For Gulf and Egyptian employers, ask this before choosing a region.
3. What traffic does it send? Requests per minute, average input and output tokens, burst shape, cache potential, from measurement where possible.
4. Which deployment type answers the question? Standard, provisioned with spillover, Priority, Flex or Batch, with the reason.
5. What is the isolation requirement? Which level, and what the failure it prevents is.
6. How does it authenticate? Managed identity or federated credential, the role, and the scope. If keys appear, why.
7. What happens when the model is retired? Who evaluates the replacement, on what data, and by when.
8. What happens when the region fails? The second deployment or region, and how failover was tested.
9. What does it cost, and who is charged? Estimate with token counts, price category and the tier, and the chargeback dimension.
10. How will we know it is broken? Alerts, dashboards, and what is logged, including whether prompt bodies are stored and for how long.
11. What would make us leave? Name the trigger for choosing another provider or a self-hosted model.

Operational recipes

Find which deployments are close to retirement

SUB=$(az account show --query id -o tsv)
LOC=eastus
az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/locations/$LOC/models?api-version=2024-10-01" \
  --query "value[].{name:model.name,version:model.version,status:model.lifecycleStatus}" -o table

Remember the inverted words in the status column. Join this to your deployment inventory from az cognitiveservices account deployment list, and alert on the dates, which sit on the per-SKU deprecation fields of each model entry.

Check capacity before you create a provisioned deployment

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/modelCapacities?api-version=2024-10-01&modelFormat=OpenAI&modelName=gpt-5.1&modelVersion=2025-11-13"

Read availableCapacity per location and SKU. A deployment succeeds, and only then does a reservation make sense.

See your quota tier

az rest --method get \
  --url "https://management.azure.com/subscriptions/$SUB/providers/Microsoft.CognitiveServices/quotaTiers?api-version=2025-10-01-preview"

This is a preview API version, so expect it to change. Viewing quota needs Cognitive Services Usages Reader at subscription scope.

Verify private DNS from inside the network

for h in openai services.ai cognitiveservices; do
  host aoai-prod-gulf.$h.azure.com
done

Every answer should be a private address. Put this in the deployment pipeline, running from a build agent inside the network, and fail on a public answer.

Classify a response in a shared client

def classify(resp):
    """Return a short outcome label for metrics, not only the HTTP status."""
    if getattr(resp, "status", None) == "incomplete":
        reason = resp.incomplete_details.reason  # for example "max_output_tokens"
        return f"incomplete:{reason}"
    for item in getattr(resp, "output", []) or []:
        if getattr(item, "status", None) == "incomplete":
            return "incomplete:item"
    return "ok"

Extend the function for Chat Completions with finish_reason == "content_filter", and parse stream error events separately. Emit the label as a metric dimension next to the team tag.

Set the retry policy once

from openai import OpenAI

client = OpenAI(
    base_url="https://aoai-prod-gulf.openai.azure.com/openai/v1/",
    api_key=token_provider,   # bearer token provider from azure-identity
    max_retries=4,            # the only retry layer; do not wrap with tenacity
    timeout=60,
)

If an application uses tenacity, set max_retries=0 on the client so the layers do not multiply.

Practice cards

Card 1. A team reports 429s at 10:02 every morning. Metrics at one minute show 40% of quota used. What do you check, and what do you change? Look at requests-per-second bursts against the RPM window, then at the output cap.

Card 2. You are told a model retires in ten weeks and its replacement is not announced. What do you do this week? Inventory callers, build the evaluation set from production traffic, test the nearest current models, and put the date on the calendar with an alert.

Card 3. A regulated team wants a guarantee that their prompts never leave the EU. Which deployment types qualify, and what is not covered by that choice? Compare Data Zone and Regional processing, then address abuse-monitoring storage and stateful Responses retention.

Card 4. Spend doubled with flat traffic. Name five causes and the metric that tests each.

Card 5. Write the Azure Policy, in words, that makes the secure posture the default. Cover keys, public access and allowed deployment SKUs.

A closing checklist

• Keys are disabled wherever possible and every exception has an owner.
• Every model has a retirement date and an alert.
• One retry layer, one output-cap default, one shared client.
• Quota allocation and deployment configuration are in code.
• Every provisioned deployment has a measured size, spillover and an owner.
• All three private names are verified from inside the network.
• Cost is attributed per team and reconciled monthly.
• Failover has been tested this quarter.

Carry that review habit forward. As with most platform work, the incidents are prevented in the pull request, long before they reach production. Related reading: LiteLLM, Langfuse and OpenTelemetry.