The Complete Helicone Guide

This is part one of three. It covers what you need to do real work with Helicone: send a language-model request through it, find that request on a dashboard, label your traffic so the numbers mean something, and switch on the gateway features that save money and absorb failures. By the end you will be able to explain where every request goes, read a cost figure without guessing, and recognise the handful of errors that catch almost every newcomer. Mid-level and Senior take the same topics further; nothing here is thrown away.

One honest note before we start, because it changes how you should use this guide. Helicone is in maintenance mode. On 3 March 2026 the company announced it was joining Mintlify, and on the same day new sign-ups on Helicone Cloud were switched off. The product still runs, security fixes and new models still ship, and no end date has been published, but you cannot simply create a fresh cloud account today. This guide is written for that reality: it teaches the ideas and the exact request format, tells you which parts you can practise on your own machine, and flags the places where older tutorials on the internet are now wrong. The skills transfer directly to any other LLM observability tool, which is part of why they are worth learning.

Each section ends with a Try it task. If you have a cloud account, do them against the cloud. If you do not, the self-hosting section shows you how to run the dashboard locally, and most tasks can be read as "what would I see" exercises. These ideas only stick once you have watched your own request appear, fail, and appear again properly.

What Helicone is, and where it stands today

Helicone is an open-source LLM observability platform with a gateway attached. Two halves of that sentence matter. Observability means that every call your application makes to a language model is recorded: what you sent, what came back, how many tokens each side used, what it cost, how long it took, and whether it succeeded. Gateway means that Helicone can sit in the path of those calls and do useful things to them on the way through, such as returning a stored answer instead of paying for a new one, retrying a failed call, or refusing a call that would exceed a limit you set.

The picture is the whole product in four boxes. Your application talks to Helicone instead of talking straight to the model provider; Helicone forwards the call, hands you the answer, and records everything about the exchange so you can look at it later.

Now the status, stated plainly because it affects your plans:

• 3 March 2026: Helicone announced it was joining Mintlify and would continue "in maintenance mode", meaning security updates, new models, and bug and performance fixes keep shipping, but the product is no longer growing new features.
• Sign-ups are disabled. The sign-up page now says so. The official quick-start still says "sign up for free", and that line is stale. If you already have an account, it keeps working.
• Self-hosting is open. The code is Apache-2.0 licensed on GitHub, and there is a Docker image you can run yourself. That is the path for a learner who has no cloud account.
• Pricing is historical. The pricing page still lists a free tier of 10,000 requests a month and paid tiers, but self-serve upgrades have been removed, so treat those numbers as information rather than a shopping list.

There is no version number to install. Helicone is a continuously updated service plus a code repository. The newest tag on GitHub is v2025.08.21-1, and the self-host Docker image is helicone/helicone-all-in-one:v2025.08.21. Both are dated August 2025. When this guide says "the current docs", it means the documentation as checked in September 2026.

Why learn a tool in maintenance mode? Three reasons. First, plenty of teams run it in production today and will for a while. Second, the concepts (proxying, async logging, sessions, custom properties, cache keys, rate-limit windows) are the same ones every alternative uses, and Helicone's header-driven design makes them unusually easy to see. Third, you will be asked about it, or about moving off it, in interviews and in jobs. Knowing both the tool and its current status is exactly what a careful engineer is expected to know.

The problem it solves, and what came before

Calling a language model is easy. The first call takes four lines of code and works. The trouble starts a week later, when something happens that those four lines cannot answer.

Your bill is higher than expected, and you do not know which feature, which user, or which prompt caused it. A customer says "the assistant gave me a strange answer yesterday afternoon", and you cannot see what was actually sent to the model or what came back. The provider starts returning rate-limit errors in the middle of a demo. Latency creeps up and you cannot tell whether it is the model, your prompt, or your own code. A test run costs real money because it re-asks the same questions every time. Each of these is a question about a request that already happened, and a bare SDK call leaves no record.

Before tools like Helicone, teams solved this in three ways, each with a cost.

Print statements and log files. You write a wrapper that logs the prompt and response. This works until you need to sum cost by user, filter by error, or show a non-engineer what happened. You end up building a small analytics product by accident, and you must remember to add the wrapper to every call site.

The provider's own dashboard. OpenAI and Anthropic show you usage, but per API key and per day, not per user, per feature, or per conversation. You cannot see individual prompts in a way you can search, and if you use several providers you must check several dashboards.

General-purpose monitoring. Tools built for web servers record request counts and latencies, but they do not understand tokens, model names, or prompt cost. The interesting number, dollars per conversation, is something you have to calculate yourself.

Helicone's answer is to make the record automatic and to attach the labels you care about with almost no code. The core trick is unusual and worth noticing now: features are switched on by HTTP headers. A header is a small named value that travels with every web request. To cache responses you add Helicone-Cache-Enabled: true. To attribute the call to a user you add Helicone-User-Id: user-123. You do not install a framework or restructure your code; you add a line.

Here is how the old way and the Helicone way compare.

An honest caveat belongs here as well. Putting a gateway in front of your model calls adds one more thing that can fail, and you are trusting a third party with your prompts. We will return to both points, because choosing between the two integration styles in the next sections is exactly a choice about that trade-off.

The mental model: a few nouns

Helicone has a small vocabulary. Learn these words and the dashboard, the docs and the error messages all become readable.

Request. One logged call to a model. Every request gets a unique ID called the Helicone-Id, a UUID such as 3f2b8c1e-.... It comes back to you in a response header named helicone-id, and you can also choose your own ID by sending Helicone-Request-Id. The ID is how you attach a rating or a score to that exact call later.

Helicone API key. A secret that identifies your organisation to Helicone. It starts with sk-helicone-. You create it in the dashboard under Settings and API Keys. This is not your OpenAI or Anthropic key, and confusing the two is the single most common beginner mistake, so hold on to the distinction.

Provider key. Your key for the model provider itself, such as an OpenAI key. With the current gateway you store it inside Helicone, in a settings page called Provider Settings, and Helicone uses it on your behalf. The docs call this BYOK, "bring your own key". You are billed by the provider, not by Helicone.

AI Gateway. Helicone's hosted endpoint at https://ai-gateway.helicone.ai. It speaks the OpenAI request format, so the standard OpenAI SDK works against it, and it can translate that format to other providers behind the scenes. You authenticate to it with your Helicone key alone.

Legacy proxy. The older way: provider-specific addresses such as https://oai.helicone.ai/v1 for OpenAI and https://anthropic.helicone.ai for Anthropic. Here you send your provider key in the usual place and add your Helicone key in a separate header. The docs now describe these as "maintained but no longer actively developed". You will meet them in existing code, which is why we cover them, but new work should start on the AI Gateway.

User. The person using your application, identified with the Helicone-User-Id header. It is whatever string you choose. It powers per-user cost and per-user rate limits.

Custom property. Any label you invent, set with a header named Helicone-Property- followed by a name. Helicone-Property-Environment: production creates a property called Environment. You can filter and group by it.

Session. A group of related requests that belong to one task, such as a conversation or one run of an agent. You group them with Helicone-Session-Id, and you can give each step a place in a tree with Helicone-Session-Path.

Cache. Stored responses that Helicone can return for a repeated identical request, so you do not pay the provider twice.

The next diagram shows how the labels ride along with one call.

There is one more idea that is easy to miss and important to get right: logging is asynchronous. Helicone returns your response first and records the log afterwards, through an internal queue. That design means a slow dashboard should not slow your users, and it is why the May 2026 incident, in which logging and the dashboard were down for about four days while the gateway kept serving requests, did not break customer apps. The logs were queued and not lost.

Two ways in: through the gateway, or beside it

Before any code, choose how Helicone connects to your application. There are exactly two styles, and everything else in the docs is a variation.

Proxy, or gateway, integration. Your application sends its model calls through Helicone. Helicone is in the path: request in, forward to the provider, response out. Because Helicone sees the request before the provider does, it can do things to it: return a cached answer, retry a failure, enforce a rate limit, block a risky prompt. You get every feature. The price is that Helicone is now on your critical path. If it were unreachable, your calls would fail unless you added your own fallback.

Async integration. Your application calls the provider directly, exactly as before, and afterwards sends a copy of the exchange to Helicone for recording. Helicone is not in the path, so it cannot slow or break your calls. The price is capability: with async logging you get observability only, with no caching, no rate limits, and no gateway retries, because those all need to happen before the provider is called.

A rule of thumb that works for a beginner: start with the gateway because it is one URL change and you get everything; move to async if your organisation insists that no third party may sit in the request path, or if you need to log calls from a model that the gateway does not reach.

Two smaller details. The async route comes in flavours: the OpenLLMetry loggers (helicone-async for Python, @helicone/async for Node) wrap the provider SDKs automatically, and the manual logger (helicone-helpers, @helicone/helpers) lets you record a call by hand, which is useful for models the automatic wrappers do not cover. And the gateway comes in two generations, the current AI Gateway and the legacy provider proxies. Everything in this guide defaults to the AI Gateway.

Getting set up: which path fits you

There are three setups, and only you know which applies. All three end the same way: you have a Helicone API key and a place where requests show up.

Path A: you already have a cloud account

Log in at https://us.helicone.ai, or https://eu.helicone.ai if your organisation was created in the EU region. Open Settings, then API Keys, and generate a key. It begins with sk-helicone-. Put it in an environment variable rather than in your code:

export HELICONE_API_KEY="sk-helicone-xxxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx"

On Windows PowerShell the equivalent is $env:HELICONE_API_KEY="sk-helicone-...", and in the old command prompt it is set HELICONE_API_KEY=sk-helicone-.... The variable lasts for that terminal window only.

Then add your provider key inside Helicone, at the Providers page (https://us.helicone.ai/providers). This step matters more than it looks. The docs' headline flow says you can "add credits" and skip provider keys, but that billing route (called pass-through billing) now requires a feature flag that most accounts do not have, since an August 2026 change. Without the flag it is denied. Bringing your own provider key works for everyone, so make it your default.

Finally install only the OpenAI SDK, since the gateway speaks its format:

pip install openai          # Python
npm install openai          # Node

Path B: you have no account, and want to learn

Run Helicone on your own machine. This is the route for a newcomer, and the section named "Running Helicone yourself" walks through it. Be aware of a documented limitation there: the local image gives you the dashboard and the log store but its documentation does not currently describe how to route model traffic through it. We will be precise about what you can and cannot do.

Path C: you are reading along only

That is fine. Every example in this guide is complete, and the expected results are described. Treat the Try it tasks as thought exercises.

Checking the setup

Whatever your path, the test of a working setup is the same. A request is sent, and it appears on the Requests page of the dashboard, which the docs say happens "within seconds". You can also check the response headers: a healthy call carries a helicone-id header. The next section does exactly this.

Your first request

Everything starts with one call. We use the AI Gateway and the standard OpenAI SDK, changing exactly two things from a normal OpenAI program: the base URL, and the key.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.getenv("HELICONE_API_KEY"),
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

print(response.choices[0].message.content)

Read it line by line, because each line is a lesson.

base_url tells the OpenAI SDK where to send requests. Normally it points at OpenAI's servers; here it points at Helicone's gateway. Nothing else in your program needs to know that anything changed. api_key is your Helicone key, not an OpenAI key, because the gateway authenticates you and uses the provider key you stored earlier to talk to the model. The model string is a plain model name. On the current gateway, a bare name like gpt-4o-mini lets Helicone pick a provider that offers it, cheapest first, and fail over to another if one is down.

The same call from the command line, which is the best way to see exactly what travels over the wire:

curl https://ai-gateway.helicone.ai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello, world!"}]}'

And the Node version:

import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "https://ai-gateway.helicone.ai",
  apiKey: process.env.HELICONE_API_KEY,
});

const response = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Hello, world!" }],
});

console.log(response.choices[0].message.content);

The gateway exposes a few endpoints you will use: POST /chat/completions for chat, POST /responses for OpenAI's newer Responses format, and GET /v1/models to list what you can route to. Some pages in the docs show a /v1 prefix on these paths. The quick-start uses the root without it, and this guide does the same.

Now verify. Open the Requests page in your dashboard. You should see one row: model, status 200, a few tokens, a cost measured in fractions of a cent, and a latency. Click it to see the full request and response. That single view, every prompt and every answer with its cost, is the thing you did not have before.

To see the request ID from code, ask curl to show headers:

curl -i https://ai-gateway.helicone.ai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi"}]}' | head -20

Look for a line beginning helicone-id: in the response headers. Save it: later you will attach a score to that exact call.

If you are maintaining older code, you will see the legacy form. It sends the provider key normally and adds the Helicone key as a header:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://oai.helicone.ai/v1",
    default_headers={"Helicone-Auth": f"Bearer {os.environ['HELICONE_API_KEY']}"},
)

Note the Bearer prefix inside the Helicone-Auth value. Forgetting it is a classic source of 401 errors on the legacy proxy. Anthropic's equivalent uses base_url="https://anthropic.helicone.ai" with the same header. Recognise this shape, keep it working if you inherit it, but write new code against the AI Gateway.

Reading the dashboard

A dashboard is only useful if you know what each number means, so spend a few minutes learning to read this one. Three areas matter for a beginner.

The Requests page. One row per call. The columns you will look at constantly are the timestamp, the model, the status code, the token counts (prompt tokens are what you sent, completion tokens are what came back), the latency, and the cost. You can filter by any of them, and by the labels you attach later, such as user and custom properties. Clicking a row opens the full request and response bodies, which is how you answer "what exactly did the model see?".

Cost. Helicone calculates cost from the usage numbers the provider returns and from its own price list, an open-source registry covering 300+ models. The figure is an estimate from public prices, not your invoice. If you negotiated pricing or use a model the registry cannot resolve, it may differ or be missing. Treat it as accurate to a few percent for ordinary models and verify large numbers against the provider's bill.

Latency. The time from your request arriving to the response finishing. For streaming responses there is also a time to first token, the delay before the first words appear, which is what users actually feel. A rising latency line can mean a slower model, longer prompts, or provider trouble, and filtering by model tells you which.

A token, for readers new to language models, is a chunk of text of roughly three or four characters. Models read and write tokens, and providers charge per token, with output tokens usually costing more than input tokens. This is why long prompts and long answers are both expensive, and why the dashboard shows the two counts separately.

Here is a realistic record to practise reading. Imagine the row says: model gpt-4o-mini, status 200, prompt tokens 42, completion tokens 18, latency 820 ms, cost $0.00001. You can tell at a glance that the call succeeded, was short, and cost a fraction of a cent. If another row says status 429 with 0 completion tokens, the call failed before generating anything, and the body will contain the provider's error message explaining why.

Naming your traffic: users, properties and sessions

A list of requests is useful; a list you can slice is powerful. Labels are how you slice, and all of them are headers you add to a call. This is where Helicone starts paying for itself.

Setting headers

There are two places to set headers. Per request, when one call needs a specific label:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarise our refund policy."}],
    extra_headers={
        "Helicone-User-Id": "user-123",
        "Helicone-Property-Feature": "refund-bot",
        "Helicone-Property-Environment": "production",
    },
)

Or once for the whole client, so every call carries them:

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.getenv("HELICONE_API_KEY"),
    default_headers={
        "Helicone-Property-Environment": "production",
        "Helicone-Property-App": "support-assistant",
    },
)

In Node the per-request form passes a second argument: client.chat.completions.create({...}, { headers: { "Helicone-User-Id": "user-123" } }), and the per-client form is defaultHeaders.

Users

Helicone-User-Id is the identifier of the person using your application. Use a stable, non-sensitive ID such as your database's user ID, not an email address or a name, because these values are stored and shown in the dashboard. With it you can see cost per user, find your heaviest users, and later limit spending per user. Note that the per-user rate limit needs this header to exist; without it there is nobody to count against.

Custom properties

A property is any label you choose. The header is Helicone-Property- followed by a name, and the value is free text. Good properties answer the questions you listed earlier: Environment (production, staging, local), Feature (which part of your product made the call), PromptVersion, Tenant for a multi-customer product. Once set, you filter and group by them on the dashboard, set alerts on them, and even rate-limit by them.

Choose property names with care. They are the columns of your future analysis, so use consistent capitalisation and a small, agreed set across the team. Environment and environment would be two separate properties.

Sessions

Many modern applications make several model calls to answer one user request: an agent plans, searches, reads and writes. A session groups those calls so you can see the whole story. Three headers work together:

import uuid

session_id = str(uuid.uuid4())

def ask(question, path):
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
        extra_headers={
            "Helicone-Session-Id": session_id,
            "Helicone-Session-Path": path,
            "Helicone-Session-Name": "Trip Planner",
        },
    )

ask("List three cities worth visiting in Jordan.", "/plan")
ask("What is the best month to visit Petra?", "/plan/timing")
ask("Write a three-day itinerary for Amman.", "/write")

Paths start with a slash and should describe what the step is, not when it ran: group by function (/research, /write), not by time. The docs list all three session headers as required together, so send all three for a proper tree. On the Sessions page you then see one session with three requests, the total cost of the run, and the tree shape, which is how you find the one expensive step in a ten-step agent.

Provider keys, model names and routing

The AI Gateway decides which provider serves your request from the model string, so it is worth understanding its small grammar. You will not need every form on day one, but you should recognise them when you read other people's code.

The suffix after the slash names the provider. This is the current form. An older README for Helicone's standalone Rust gateway used a prefix (openai/gpt-4o-mini) and a different base URL ending in /ai. That project has been abandoned, so if you see the prefix form in a tutorial, it is from a previous generation, and mixing the two forms will fail.

Why does routing matter to a beginner? Because it explains two behaviours you will otherwise find mysterious. First, the same model name can be served by different providers on different calls, which shows in the dashboard's provider column. Second, automatic failover means a provider outage may be invisible to your users, at the cost of your calls sometimes landing on a provider you did not expect. If that matters for compliance or data residency, pin the provider with the suffix.

On the question of whose key is used: the docs say the gateway tries your own keys before Helicone-managed credits, but a second docs page says the opposite order. The two pages disagree, so do not build a design that depends on the order. The safe beginner setup is the simple one: store your own provider keys, use them, and ignore credits.

For models that Helicone's registry does not know, routing goes only through your own stored deployments. This is the situation with a custom Azure deployment or a fine-tuned model, and it is another reason to add your provider keys early.

Caching: stop paying for the same answer twice

A model answers the same question the same way often enough that repeating the call wastes money. Think of a support bot that gets "what are your opening hours?" a hundred times a day, or of your own test suite that re-runs identical prompts on every commit. Caching stores the first response and returns it for later identical requests, instantly and at no provider cost.

It is off by default. You switch it on with one header:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What are your opening hours?"}],
    extra_headers={"Helicone-Cache-Enabled": "true"},
)

Send that same request twice. The first is a miss and goes to the provider. The second returns from the cache. To check which one you got, read the response header Helicone-Cache, which says HIT or MISS.

The cache is tunable with a few more headers:

Understand how a hit is decided, because it explains nearly every caching surprise. Helicone builds a cache key by hashing the seed, the full URL, the full request body, and some headers including the authorisation. Two requests hit the same entry only if all of that is identical. One changed character in the prompt, one different temperature, or one timestamp buried in the body gives a different key, and therefore a miss.

The bucket setting deserves a sentence. With a bucket size of one, the same question always returns the same stored answer, which is what you want for deterministic things. With a larger bucket, Helicone stores several answers to the question and returns one at random, preserving some of the variety you would get from a live model.

Where do cached answers live? On Helicone's edge network, in Cloudflare's key-value storage, not in your own infrastructure. For a team with data-residency rules, such as many Gulf and Egyptian employers with regulated customer data, that is a fact to raise with whoever owns compliance before caching prompts that contain personal data.

Retries and rate limits

Two more gateway features protect you from failures and from yourself.

Retries

Model providers sometimes fail briefly: a 429 because they are busy, a 503 during a hiccup. Retrying after a short wait usually succeeds. Helicone can do that for you:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about the Nile."}],
    extra_headers={
        "Helicone-Retry-Enabled": "true",
        "Helicone-Retry-Num": "3",
        "Helicone-Retry-Factor": "2",
    },
)

Helicone-Retry-Num is how many retries (default 5), Helicone-Retry-Factor is the multiplier for exponential backoff, meaning each wait is that many times longer than the last (default 2), and the minimum and maximum waits default to 1000 and 10000 milliseconds. Retries happen on status 429, 500, 502, 503 and 504, and never on other 4xx errors, which would fail again identically: a malformed request does not fix itself by being repeated. Each attempt is logged separately, so you can see the retries in the dashboard.

The docs add a useful piece of guidance: if you use the gateway's automatic provider routing, failover to another provider is usually better than retrying the same one. Retries earn their place when you have pinned a single provider.

Rate limits

A rate limit caps how much traffic passes in a window of time. The use is protection: a runaway loop, an abusive user, or a bug that would otherwise spend your month's budget in an hour. The header has a compact format:

Helicone-Rate-Limit-Policy format
Helicone-RateLimit-Policy: [quota];w=[seconds];u=[request|cents];s=[user|property]

Read it piece by piece. quota is the allowed amount. w is the window length in seconds, and it must be at least 60. u is the unit, either request (the default) or cents for a spending cap. s is the segment, meaning what to count separately, such as per user or per property value.

# 1000 requests per hour, across everyone
extra = {"Helicone-RateLimit-Policy": "1000;w=3600"}

# 100 requests per day, per user (needs the user header)
extra = {
    "Helicone-RateLimit-Policy": "100;w=86400;s=user",
    "Helicone-User-Id": "user-123",
}

# 500 cents ($5) per hour, per user
extra = {
    "Helicone-RateLimit-Policy": "500;w=3600;u=cents;s=user",
    "Helicone-User-Id": "user-123",
}

When a call exceeds the policy, Helicone returns 429, and the response headers Helicone-RateLimit-Limit and Helicone-RateLimit-Remaining tell you the state. Your code should treat that 429 like any other: back off, show a friendly message, or queue the work.

Both features live in the gateway, so both need proxy integration. If you chose async logging, neither exists for you, and you must implement retries and limits in your own code.

Feedback, scores and finding requests later

Observability is not only watching; it is also judging. Was the answer any good? Helicone lets you attach a verdict to a request after the fact, and then query your history programmatically.

The glue is the request ID. Either read it from the helicone-id response header, or choose your own by sending Helicone-Request-Id with a UUID you generate, which is easier because you know the ID before the call returns.

Feedback is a thumbs-up or thumbs-down from a person, stored as a boolean. Scores are named numbers or booleans from any source, for example an automated check. Both go to the REST API at https://api.helicone.ai (the EU region uses https://eu.api.helicone.ai), authenticated with your Helicone key as a bearer token:

# A user clicked thumbs-up
curl -X POST "https://api.helicone.ai/v1/request/$REQUEST_ID/feedback" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rating": true}'

# An automated check produced scores
curl -X POST "https://api.helicone.ai/v1/request/$REQUEST_ID/score" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scores": {"accuracy": 92, "is_safe": true}}'

There is a trap in the second call. Scores must be integers or booleans. A float such as 0.92 is rejected. The convention is to multiply by 100 and send 92. Scores are also aggregated with a delay of roughly ten minutes, so do not panic if they do not appear instantly.

You can add a property to an existing request as well, with PUT /v1/request/{requestId}/property and a body like {"Environment": "production"}.

To pull history out, the request-query endpoint accepts a filter:

curl -X POST "https://api.helicone.ai/v1/request/query-clickhouse" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "request_response_rmt": {
        "properties": { "Environment": { "equals": "production" } }
      }
    },
    "limit": 10,
    "offset": 0
  }'

Study the filter shape, because it is the classic way to get an empty list. The property condition must be wrapped inside request_response_rmt, which is the name of the underlying table. If you write the property filter at the top level, the API does not complain: it returns [], and you will wonder where your data went. Conditions combine with {"left": ..., "operator": "and", "right": ...} when you need more than one.

Streaming, privacy and the regions

Three smaller topics that beginners bump into in their first week.

Streaming and cost

Chat applications stream: the model's words arrive a few at a time instead of all at once. The cost of a streamed response depends on the usage figures the provider sends at the end of the stream, and by default some providers do not send them. The result is a cost of zero or a wrong number in the dashboard. The fix is to ask for usage explicitly:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Tell me about Cairo in two sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

If you cannot change the call, the header helicone-stream-usage: "true" does the same job. Forgetting this and then trusting the streamed cost column is an easy way to under-report your spend.

Keeping bodies out of the logs

Sometimes a prompt contains something you do not want stored: a national ID, a medical note, a password someone typed into a chat box. Two headers tell Helicone not to keep the bodies:

extra_headers={
    "Helicone-Omit-Request": "true",
    "Helicone-Omit-Response": "true",
}

Read the fine print. These stop the bodies being stored, but they are still sent to Helicone's backend and held briefly in memory. They reduce what is kept; they do not keep data from ever reaching Helicone. If a regulation forbids your data leaving your network, these headers are not sufficient, and self-hosting or async logging with content tracing disabled is the conversation to have. Redacting sensitive values in your own code before the call is always the strongest protection, and the simplest.

Regions

Helicone Cloud has a US region (us.helicone.ai) and an EU region (eu.helicone.ai), and your dashboard and REST API live in whichever your organisation was created in. The EU region uses https://eu.api.helicone.ai for the REST API, and forgetting this swap makes correct keys return authentication errors. There is no Middle East region, so teams whose customers' data must stay in the Gulf or in Egypt should treat the choice between EU cloud, self-hosting, or a different tool as a compliance decision, not a technical afterthought.

Running Helicone yourself

If you have no cloud account, this is how you get a real Helicone to play with. The project publishes one Docker image that bundles the dashboard, the API service, the databases and the file storage, so a single command starts everything. If you are new to containers, the Docker guide explains what the commands below do.

docker pull helicone/helicone-all-in-one:latest

docker run -d \
  --name helicone \
  -p 3000:3000 \
  -p 8585:8585 \
  -p 9080:9080 \
  helicone/helicone-all-in-one:latest

Three ports are published: 3000 is the dashboard, 8585 is the API service (called Jawn), and 9080 is the object storage where request bodies live. Open http://localhost:3000 in a browser. To confirm the API service is alive, run:

curl http://localhost:8585/healthcheck

There is a wrinkle that catches everyone. The container has no email service, so after you register an account, the account is unverified and you cannot log in. You verify it by hand, directly in the container's Postgres database:

docker exec -u postgres helicone psql -d helicone_test -c \
  "UPDATE \"user\" SET \"emailVerified\" = true WHERE email = 'your@email.com';"

If the dashboard then says "No organization ID found", your user exists but belongs to no organisation, and you must create one and add yourself to it with the SQL in the official self-host page. The official page lists the exact statements; copy them from there rather than retyping.

The second wrinkle: data vanishes when the container is removed, because the databases live inside it. To keep your data, mount volumes:

docker run -d --name helicone \
  -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -v helicone-postgres:/var/lib/postgresql/data \
  -v helicone-clickhouse:/var/lib/clickhouse \
  -v helicone-minio:/data \
  helicone/helicone-all-in-one:latest

Now the honest part about what you can do with this. The all-in-one image runs the dashboard and the API service. It does not include the LLM gateway: in August 2026 the API service stopped proxying model traffic, and requests to its old /v1/gateway/... routes now return 404. The documentation says model traffic should go through "the AI Gateway, which you can deploy next to this image", but the self-hosted gateway page currently returns Not Found, and the standalone Rust gateway project is abandoned. The documented path for self-hosters to get traffic in is therefore logging rather than proxying, and the exact endpoint configuration for pointing the logging SDKs at your own server is not spelled out in the docs. For a beginner this means a local install is excellent for learning the dashboard, the data model and the operations, but do not expect the gateway features (caching, retries, rate limits) to work against it.

If you ever run it on a remote server, every URL setting must share one origin. The dashboard URL, the API URL and the storage URL must all be the same host, because mixing localhost with a public IP produces an "Invalid origin" error at sign-in. The variables involved are SITE_URL, BETTER_AUTH_URL, NEXT_PUBLIC_APP_URL, NEXT_PUBLIC_HELICONE_JAWN_SERVICE and S3_ENDPOINT, and NEXT_PUBLIC_IS_ON_PREM must be set to true or you meet an infinite redirect loop. Mid-level and Senior cover production self-hosting.

Configuration and the errors you will actually meet

By now you have seen most of Helicone's configuration. It is almost all headers, plus two secrets. It helps to hold the whole surface in one table.

Reading the errors

When something fails, work in this order: read the status code, then read the error message body, then check the dashboard row if one exists. A request that reached Helicone and failed is still logged, which is itself useful evidence.

401 "Authentication failed", type invalid_api_key. The gateway tried to call the provider and the provider key stored in Helicone was rejected. Open the Providers page and re-enter the key. If you are on a legacy URL, the cause is usually the key mix-up: provider key in Authorization, Helicone key in Helicone-Auth: Bearer sk-helicone-..., with the Bearer prefix.

429 Insufficient credits. You have no provider key stored, or no credits, so the gateway could not find a way to pay for the call. Add a provider key.

429 with Helicone-RateLimit-Remaining: 0. This is your own policy working. Wait for the window or change the policy.

403 "Wallet suspended" or a blocked model. An account-level block. Email support; nothing in your code causes it.

400 PROMPT_FLAGGED_FOR_MODERATION or PROMPT_THREAT_DETECTED. You or an admin enabled the optional moderation or security screening headers and a prompt was blocked. Handle these errors in your app as an expected outcome, not a crash.

The cache always says MISS. Compare two requests for any changing field, and send header values as strings.

Retry or other headers seem ignored. A value was sent as a number or boolean. Make it a string.

Query API returns []. The property filter is not wrapped in request_response_rmt.

A score is rejected. It was a float. Send an integer or a boolean.

Streaming cost is zero or off. No usage chunk. Add stream_options={"include_usage": True}.

Cost missing for a custom model. Helicone cannot resolve the model name to a price. The Helicone-Model-Override header supplies a name for cost calculation.

Self-host: the browser shows "connection refused" for API calls. The page is calling localhost:8585 from a machine where that is wrong. Set NEXT_PUBLIC_HELICONE_JAWN_SERVICE to the address browsers can reach.

Self-host: "Invalid origin" or an endless redirect. The origin variables disagree, or NEXT_PUBLIC_IS_ON_PREM=true is missing.

Self-host: "/v1/gateway" returns 404. That route was removed in August 2026. It is not misconfigured.

Sign-up page says "Sign ups are disabled". That is maintenance mode. Self-host instead.

Putting it all together

Let us build one small project that uses everything above: a command-line assistant that answers questions for a fictional customer-support desk, with labelled traffic, a cache, retries, a rate limit, a session, feedback and a spending check. Save it as desk.py. It assumes HELICONE_API_KEY is set and a provider key is stored in Helicone.

import os
import sys
import uuid
import requests
from openai import OpenAI

HELICONE_KEY = os.environ["HELICONE_API_KEY"]

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=HELICONE_KEY,
    default_headers={
        "Helicone-Property-Environment": "local",
        "Helicone-Property-App": "support-desk",
        "Helicone-Cache-Enabled": "true",
        "Helicone-Retry-Enabled": "true",
        "Helicone-Retry-Num": "3",
    },
)

session_id = str(uuid.uuid4())


def ask(user_id: str, question: str) -> tuple[str, str]:
    request_id = str(uuid.uuid4())
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a concise support agent."},
            {"role": "user", "content": question},
        ],
        extra_headers={
            "Helicone-Request-Id": request_id,
            "Helicone-User-Id": user_id,
            "Helicone-Session-Id": session_id,
            "Helicone-Session-Path": "/answer",
            "Helicone-Session-Name": "Support Desk",
            "Helicone-RateLimit-Policy": "20;w=60;s=user",
        },
    )
    return request_id, response.choices[0].message.content


def rate(request_id: str, good: bool) -> None:
    requests.post(
        f"https://api.helicone.ai/v1/request/{request_id}/feedback",
        headers={"Authorization": f"Bearer {HELICONE_KEY}"},
        json={"rating": good},
        timeout=10,
    )


if __name__ == "__main__":
    user = sys.argv[1] if len(sys.argv) > 1 else "demo-user"
    while True:
        question = input("Question (blank to quit): ").strip()
        if not question:
            break
        rid, answer = ask(user, question)
        print(answer)
        verdict = input("Helpful? [y/n]: ").strip().lower()
        rate(rid, verdict == "y")

Run it, ask the same question twice, rate one answer up and one down, and ask five different questions. Then open the dashboard and inspect the results in this order.

1. Requests page, filtered by the property App = support-desk. You should see all your calls and only yours.
2. The repeated question. The second copy should be a cache hit with near-zero cost and latency.
3. Sessions page, where "Support Desk" shows one session containing every call, with a total cost.
4. One request, opened. It shows the user ID, the properties, and your feedback rating.
5. Filter by user to see the spending of demo-user.

Then answer the questions the project was built to answer: which question cost the most, what share of calls were cache hits, and what was the session's total cost. If you can answer those three from the dashboard in under a minute, you have achieved what a bare SDK call could not.

If you are on the local self-hosted install without a gateway, read this project as a design exercise: the labelled headers and the feedback call are the same ideas, but the cache, retry and rate-limit headers need a gateway to act on them.

What you can now do, and what comes next

You can now explain what Helicone is and where it stands: an Apache-2.0 LLM observability tool with a gateway, in maintenance mode since March 2026, with sign-ups closed and self-hosting open. You can distinguish the two integration styles and say what each costs you. You can send a request through the AI Gateway with the OpenAI SDK, find it on the dashboard, and read its tokens, cost and latency. You can label traffic with users, properties and sessions, switch on caching, retries and rate limits with string-valued headers, attach feedback and scores to a request, query history with the correct filter shape, cope with streaming cost, and keep bodies out of storage with a clear understanding of the limits of that. You can run the dashboard locally and explain honestly what a local install does and does not give you. And you can read the common errors by their status code.

That is a solid foundation, and it is also the set of things people get wrong in interviews. The interview guide for this level turns it into questions and model answers, and the tips guide collects the mistakes that cost beginners the most time.

What comes next, at Mid-level: prompt management with versioned templates and environments, provider routing and fallback chains in depth, async logging for custom models, alerts and webhooks, the query language, and how to wire Helicone into an agent framework. At Senior: architecture and failure modes, security and the trust model, data export and migration plans, multi-tenant attribution, upgrades when there are no maintained releases, and how to argue for or against keeping a tool in maintenance mode.

Two neighbouring guides in this catalogue are worth reading alongside. Langfuse is an observability platform with a similar purpose and is a frequent migration target, and Docker is the foundation for the self-hosting steps. Reading them next to this one shows you which ideas are Helicone's and which are simply how LLM observability works.

Sources

• Helicone documentation home
• Quick start
• Platform overview
• AI Gateway overview
• Provider routing
• Gateway error handling
• Header directory
• Caching
• Custom rate limits
• Retries
• Custom properties
• User metrics
• Scores
• Feedback
• Omit logs
• Sessions
• Proxy versus async
• How Helicone calculates cost
• Enable stream usage
• Query requests (ClickHouse) REST reference
• Self-host overview
• Self-host with Docker
• Joining Mintlify (announcement)
• AWS account incident write-up
• Pricing
• Helicone on GitHub
• helicone-all-in-one on Docker Hub

This page is for the conversation, not the keyboard. It assumes you have read the detailed guide or used Helicone once, and it gives you the answers an interviewer at the beginner level is listening for: what the tool is, how a request travels, which commands and headers to have ready, and which traps catch people and why. Read it out loud. Interview answers that live only in your head tend to fall apart on the first follow-up.

The thirty-second answer

Helicone is an open-source observability platform and gateway for language-model calls. You point your application's model requests at Helicone, or ship logs to it, and it records every request with its tokens, cost, latency and status, lets you label traffic by user, session and custom property, and, when you route through its gateway, adds caching, retries and rate limits switched on with HTTP headers.

The part that marks you as current: it has been in maintenance mode since it joined Mintlify in March 2026, new cloud sign-ups are disabled, and self-hosting is the open path for new users. If you say that unprompted, the interviewer knows you have looked at the tool as it is today rather than as a 2024 blog post described it.

The model

Hold five ideas and you can answer most questions.

One. Two integration styles. Proxy (gateway) integration sends your calls through Helicone, so it can act on them: cache, retry, rate-limit, route. Async integration calls the provider directly and ships a copy to Helicone afterwards, so it cannot slow your calls but also cannot cache, retry or limit them. The trade is capability against critical-path risk.

Two. Features are headers. Nothing is configured in code beyond the base URL. Helicone-Cache-Enabled, Helicone-User-Id, Helicone-Property-Feature, Helicone-Session-Id, Helicone-Retry-Enabled and Helicone-RateLimit-Policy are ordinary HTTP headers, and every value is a string.

Three. Two kinds of key. A Helicone API key (prefix sk-helicone-) identifies your organisation to Helicone. A provider key is your OpenAI or Anthropic key. On the current AI Gateway, only the Helicone key goes in Authorization, and the provider key is stored in Helicone's Provider Settings (bring your own key, BYOK). On the legacy proxies, the provider key goes in Authorization and the Helicone key goes in Helicone-Auth: Bearer ....

Four. Labels decide what you can ask later. Users, properties and sessions are attached at request time. A request logged without a label cannot be sliced by it afterwards.

Five. Logging is asynchronous. The response returns first and the log follows through a queue. That is why a logging outage does not stop the gateway from serving, as the company's own May 2026 incident showed.

The two generations of gateway come up often enough to have their own short comparison.

The status question

Expect this to be asked directly, in some form: "Would you adopt Helicone for a new project today?" It is a judgement question and there is no single right answer, but there is a well-structured one.

State the facts: acquired by Mintlify on 3 March 2026, maintenance mode (security fixes, new models and bug fixes continue, no new features), sign-ups disabled the same day, no published end-of-life date, self-serve upgrades removed in August 2026. Then reason: for a new project with no existing account, you cannot use the cloud at all, so the real options are to self-host a codebase that is no longer growing, or to choose an actively developed alternative. For an existing deployment, the sensible stance is to keep it running, keep your integration thin (one base URL and a set of headers) so it can be swapped, export your data periodically, and have a migration plan.

A strong answer says that the skills transfer. Sessions, properties, cache keys and rate-limit windows exist in every comparable tool, so learning them through Helicone is not wasted.

The commands and headers to have ready

You are unlikely to be asked to type these from memory, but being able to sketch them shows you have used the tool.

The first request through the AI Gateway:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.getenv("HELICONE_API_KEY"),
)

r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

The same call with curl, the thing to reach for when debugging:

curl https://ai-gateway.helicone.ai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello, world!"}]}'

Labels and features in one call:

extra_headers = {
    "Helicone-User-Id": "user-123",
    "Helicone-Property-Environment": "production",
    "Helicone-Session-Id": session_id,
    "Helicone-Session-Path": "/plan",
    "Helicone-Session-Name": "Trip Planner",
    "Helicone-Cache-Enabled": "true",
    "Helicone-Retry-Enabled": "true",
    "Helicone-RateLimit-Policy": "100;w=86400;s=user",
}

The headers worth reciting, in the order an interviewer might probe them:

And the REST calls for after-the-fact judgements, which use https://api.helicone.ai (EU: https://eu.api.helicone.ai) and the Helicone key as a bearer token:

curl -X POST "https://api.helicone.ai/v1/request/$REQUEST_ID/feedback" \
  -H "Authorization: Bearer $HELICONE_API_KEY" -H "Content-Type: application/json" \
  -d '{"rating": true}'

curl -X POST "https://api.helicone.ai/v1/request/$REQUEST_ID/score" \
  -H "Authorization: Bearer $HELICONE_API_KEY" -H "Content-Type: application/json" \
  -d '{"scores": {"accuracy": 92, "is_safe": true}}'

And the self-host commands, since "how would you try it without an account" is a natural follow-up:

docker run -d --name helicone \
  -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -v helicone-postgres:/var/lib/postgresql/data \
  -v helicone-clickhouse:/var/lib/clickhouse \
  -v helicone-minio:/data \
  helicone/helicone-all-in-one:latest

curl http://localhost:8585/healthcheck

Request flow, in one breath

Interviewers love "walk me through what happens when my app makes a call". Have a clean version.

Your code sends an OpenAI-format request to the gateway with your Helicone key and any feature headers. The gateway, running at the edge on Cloudflare's network, checks the feature headers: if caching is on and there is a stored answer for that key, it returns it straight away; if a rate-limit policy applies, it checks the counter and may return 429; otherwise it resolves the model string to a provider, calls that provider with the key you stored, and retries or fails over if the call fails in a retryable way. The response goes back to your app. Only after that is the log written, asynchronously, through a queue into the analytics store for metadata and object storage for the full bodies, and the dashboard reads from those.

Add one sentence on why the order matters: the log is written after the response, so observability stays off the latency path even in the proxy mode.

Cost and tokens

How does Helicone know what a call cost? It reads the token counts the provider reports in the response's usage data and multiplies by a maintained, open-source price list for 300 or more models. The figure is an estimate from public prices, not an invoice.

Why can streamed responses show the wrong cost? Providers often send usage only if asked. Without stream_options={"include_usage": True} or the header helicone-stream-usage: "true", the usage figures are missing and the cost comes out wrong or zero.

Why does cost depend on input and output separately? Providers price them differently, usually with output tokens more expensive, so the dashboard shows prompt and completion tokens as separate columns.

The classic traps, and the one cause behind most of them

Most Helicone mistakes share a single root: a setting that fails silently instead of loudly. A header sent with the wrong type does nothing and raises no error. A filter in the wrong shape returns an empty list and no complaint. A limit without its matching header counts nothing. Knowing that pattern lets you debug by asking "what should have been logged, and is it there?" rather than hunting for an exception that will never come.

The traps, each with the reason it happens:

Non-string header values. Client libraries may drop or mangle a real boolean or number in a header. "true" and "3" work; True and 3 may not. The feature simply stays off.

Mixing up the keys. The two authentication models differ between the gateway and the legacy proxy. Sending the provider key where the Helicone key belongs, or forgetting the Bearer prefix in Helicone-Auth, gives a 401.

A cache that never hits. The cache key is a hash of the whole body, so a timestamp, a random ID, or a changed temperature makes every request unique. Helicone-Cache-Ignore-Keys exists to exclude such fields.

A per-user limit that never triggers. s=user needs Helicone-User-Id on the same request, or there is nothing to count.

An empty query result. Property filters must be wrapped in request_response_rmt. Without it the API returns [].

A rejected score. Scores are integers or booleans; 0.92 fails, 92 works.

Zero cost on streams. No usage chunk, see above.

Following a stale tutorial. Legacy proxy URLs, the openai/gpt-4o-mini prefix format, Supabase-era self-hosting and the Experiments feature are all out of date.

Data gone after a restart. The self-hosted container keeps data inside itself unless you mount volumes.

Assuming the omit headers keep data private. They stop storage, not transmission.

Gateway versus async, in interview form

This distinction is the most reliable beginner-to-mid discriminator, so prepare a crisp version.

Gateway mode puts Helicone on your request path. Its strengths are every feature that must act before the provider is called, meaning caching, retries, rate limits, routing and fallback. Its risks are an added network hop and a dependency: if the gateway is unreachable, calls fail unless you have a fallback path.

Async mode calls the provider directly and sends logs afterwards through the OpenLLMetry loggers or the manual logger. Its strengths are zero impact on latency and availability, and it is the right choice when a policy forbids third-party proxies. Its cost is that you lose all the gateway features and must implement them yourself.

The sentence that shows maturity: "I would ask whether the product must keep answering users if the observability vendor is down. If yes, async, or a gateway with a direct-call fallback."

Common interview questions

What problem does Helicone solve? A bare model call leaves no record, so you cannot answer cost, quality or debugging questions after the fact. Helicone records every request with tokens, cost, latency and status, lets you slice by user, session and custom property, and can cache, retry and rate-limit through the gateway.

What are the two ways to integrate, and how do you choose? Proxy or gateway, where calls go through Helicone and every feature works, and async, where calls go direct and logs are shipped afterwards. Choose the gateway for capability and simplicity; choose async when Helicone must stay off the critical path or a policy forbids a proxy.

How do you turn on a feature like caching? With a request header: Helicone-Cache-Enabled: "true". Headers can be set per request with extra_headers or for every call with default_headers. All values are strings.

What is the difference between the Helicone key and a provider key? The Helicone key authenticates you to Helicone. The provider key authenticates Helicone, or you, to OpenAI or Anthropic. On the AI Gateway only the Helicone key is sent, and provider keys live in Helicone's Provider Settings.

How do you track cost per user? Send Helicone-User-Id with every call and use the per-user views or filters. For a per-user spending cap, add Helicone-RateLimit-Policy: "500;w=3600;u=cents;s=user" alongside the user header.

How does a cache hit get decided? Helicone hashes the seed, URL, full body and relevant headers into a key. Identical key means a hit. Anything that varies, such as a timestamp in the body, causes a miss, and Helicone-Cache-Ignore-Keys lets you exclude fields.

What do retries cover, and what do they not? Status 429, 500, 502, 503 and 504, with exponential backoff, defaults of 5 attempts. They never retry other 4xx errors, which would fail again identically. When using automatic routing, failover to another provider is usually better than retrying one.

What is a session, and why use one? A group of related requests, such as one agent run, joined by Helicone-Session-Id. The path header places each call in a tree. You use it to see total cost and find the expensive or slow step in a multi-call task.

How do you attach a quality signal to a request? Capture the helicone-id response header, or set your own with Helicone-Request-Id, then post feedback (a boolean) or scores (integers or booleans) to the REST API for that ID.

Why is a cache answer not a privacy-neutral thing? Cached responses live in Cloudflare's edge storage, not in your infrastructure, so caching prompts that contain personal data is a data-residency decision.

How would you try Helicone with no cloud account? Sign-ups are disabled, so self-host the all-in-one Docker image, mounting volumes for the three data stores. Note that it gives you the dashboard and log store, but the documentation does not currently describe a self-hosted LLM gateway, so gateway features are not something to expect locally.

What changed recently that an older tutorial would get wrong? The AI Gateway replaced provider-specific proxies as the recommended route; model strings use a suffix for the provider; self-hosting moved from Supabase to Better Auth; prompts use prompt_id and inputs; Experiments were removed in August 2026; and pass-through credits now need a feature flag, so BYOK is the practical default.

What would you do if the dashboard showed requests but cost was zero? Check whether they are streamed calls without the usage option, and whether the model name resolves to a known price. Helicone-Model-Override gives a name for cost mapping when it does not.

A teammate says "Helicone is down, is our app down?" How do you answer? It depends on the integration. With the gateway, the proxy is on the critical path, so you need to know whether the gateway or only the logging is affected; in the May 2026 incident logging and the dashboard were down for about four days while the proxy kept working. With async logging, the app is unaffected and only the records are delayed.

Why might a rate limit not be enforced? The policy's segment needs a matching header: s=user needs Helicone-User-Id, and s=<property> needs the matching Helicone-Property- header. The window must also be at least 60 seconds.

What would you check first when a request returns 401? Which integration style it is. On the gateway, the provider key stored in Helicone may be invalid, or the wrong key was used as the API key. On the legacy proxy, the Helicone-Auth header may be missing or without the Bearer prefix.

Questions to ask back

Good candidates ask questions. At the beginner level, two or three are plenty.

• "Are we on the AI Gateway or the legacy proxies, and is there a plan to move?" It shows you know there are two generations.
• "What is our plan for the maintenance-mode status: do we export data regularly?" It shows you think about dependency risk.
• "Which labels do we attach to requests today?" It shows you understand that the value of the data depends on the labels.

Sixty-second self-test

Answer each aloud without looking above, then check.

1. In two sentences, what is Helicone, and what is its current status?
2. Name the two integration styles and one thing only the first can do.
3. Which key goes in Authorization on the AI Gateway? On the legacy proxy?
4. What type must every header value be?
5. Which header attributes a call to an end user, and which makes a custom label?
6. Give two reasons a cache always misses.
7. Why might a per-user rate limit never trigger?
8. What shape must a property filter have in the query API?
9. Why might a streamed call show zero cost, and what fixes it?
10. What do the omit headers not do?
11. Why can a new learner not simply sign up, and what is the alternative?
12. Why does a logging outage not necessarily stop your app?

If you stumbled on more than three, reread the matching sections of the detailed guide and try the self-test again tomorrow.

These are the mistakes beginners actually make with Helicone, written as symptom, cause and fix, followed by the habits that prevent them and a checklist to keep. Most of the pain has one flavour: a setting that does not work and does not tell you. Learning to recognise that flavour is worth more than memorising any single fix.

Common errors at this level

"I get a 401 and I pasted my key correctly"

Symptom. The first call returns 401, often with "Authentication failed" and "type": "invalid_api_key".

Cause. On the current AI Gateway there are two keys in play. The Helicone key is what you send. The provider key (OpenAI, Anthropic) is stored inside Helicone, on the Providers page, and the gateway uses it to call the model. A 401 from the gateway is most often that stored provider key being wrong or missing, not the Helicone key you pasted. On the legacy proxies the cause is different: the provider key was sent in the Helicone slot, or the Helicone-Auth value lacks the Bearer prefix.

Fix. Open the Providers page and re-enter the provider key. If you are on a legacy URL, check the shape: provider key in Authorization, and Helicone-Auth: Bearer sk-helicone-....

"Insufficient credits" on a brand-new setup

Symptom. A 429 with the message "Insufficient credits" on your very first request.

Cause. The docs' headline flow says to add credits and skip provider keys. Since August 2026 that pass-through billing needs a feature flag on the organisation, and most accounts do not have it. With no provider key stored, the gateway has no way to pay for the call.

Fix. Add your own provider key on the Providers page. Bringing your own key works for everyone and is the route this guide recommends.

"My header does nothing"

Symptom. You add Helicone-Retry-Enabled or Helicone-Cache-Enabled, no error appears, and the feature never engages.

Cause. Header values must be strings. A Python True or an integer 3 may be dropped or converted in a way the gateway does not read, and nothing reports the problem.

Fix. Write "true" and "3". When a header seems dead, print the exact headers your code is sending, or reproduce the call with curl -i where every value is text by construction.

"The cache never hits"

Symptom. Every response header says Helicone-Cache: MISS.

Cause. The cache key is a hash of the seed, URL, full body and relevant headers. Anything that changes between calls changes the key: a timestamp or ID in the prompt, a different temperature, a different seed, or a different authorisation. Often the culprit is a line in your code that quietly adds the current time to the prompt.

Fix. Print two request bodies and diff them. Remove the varying field, or list it in Helicone-Cache-Ignore-Keys. Keep the seed constant unless you mean to separate users.

"The rate limit is not limiting"

Symptom. You set a per-user policy and a single user sends hundreds of requests without a 429.

Cause. s=user counts per value of Helicone-User-Id. If that header is missing from the same request, there is no user to count against. The same applies to s=<property> without the matching Helicone-Property- header. Also, the window w must be at least 60 seconds.

Fix. Send the segment header on every request that carries the policy. Put both in your client defaults so they cannot be separated.

"The query API returns an empty list"

Symptom. query-clickhouse answers [] although you can see matching rows in the dashboard.

Cause. A property filter has to be wrapped inside request_response_rmt. A filter at the top level is accepted and matches nothing.

Fix. Use this shape:

{
  "filter": {
    "request_response_rmt": {
      "properties": { "Environment": { "equals": "production" } }
    }
  },
  "limit": 10,
  "offset": 0
}

"My score was rejected"

Symptom. Posting {"scores": {"accuracy": 0.92}} fails.

Cause. Scores accept integers and booleans only.

Fix. Multiply by 100 and send 92, or use a boolean. Remember that scores take around ten minutes to be aggregated on the dashboard.

"Streamed calls show zero or wrong cost"

Symptom. Non-streaming requests have sensible costs, streaming requests show zero.

Cause. The provider did not send usage figures at the end of the stream, so Helicone had nothing to price.

Fix. Pass stream_options={"include_usage": True} or send the header helicone-stream-usage: "true".

"I followed a tutorial and nothing matches the docs"

Symptom. URLs, model names or menus do not match what you see.

Cause. Most Helicone tutorials predate the AI Gateway. They show oai.helicone.ai as the main route, a provider/model prefix format, Supabase-era self-hosting at port 8989, or an Experiments feature that was removed in August 2026.

Fix. Check the date on the tutorial, then confirm against the official docs listed in the detailed guide's Sources. If a page disagrees with itself, trust the current behaviour you can observe with curl.

"I cannot sign up"

Symptom. The sign-up page says "Sign ups are disabled."

Cause. That is correct. New sign-ups were switched off on 3 March 2026 when the product moved to maintenance mode.

Fix. Self-host the all-in-one image to learn the dashboard, or use an existing account. Do not spend time looking for a workaround.

"My local Helicone lost everything"

Symptom. After removing and recreating the container, your account and logs are gone.

Cause. The databases live inside the container. Without mounted volumes they disappear with it.

Fix. Start it with the three named volumes.

docker run -d --name helicone \
  -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -v helicone-postgres:/var/lib/postgresql/data \
  -v helicone-clickhouse:/var/lib/clickhouse \
  -v helicone-minio:/data \
  helicone/helicone-all-in-one:latest

"I cannot log in to my local install"

Symptom. Registration works but login fails, or the dashboard says "No organization ID found".

Cause. The container has no email service, so accounts are never verified, and a new user belongs to no organisation.

Fix. Mark the email verified with the SQL in the official self-host page, and create an organisation and membership if the second message appears. Copy the statements from the docs rather than retyping them, because the quoting is fiddly.

"Gateway routes return 404 on my self-hosted port 8585"

Symptom. Calls to /v1/gateway/... on the local API return 404.

Cause. The API service stopped proxying model traffic in August 2026. Nothing is broken; the route is gone.

Fix. Do not expect proxy features from the local container. Use the local install for the dashboard and data, and use a cloud account or another gateway for model traffic.

The habits that pay off most

Ordered by how much trouble each prevents.

1. Keep the integration thin. Put the base URL and the Helicone key in configuration, not scattered through code. One place to change means you can point at the gateway, at a different region, or away from Helicone entirely without a rewrite. Given the tool's status, this is the most valuable habit of all.

2. Set your labels on day one. Decide three or four properties (Environment, Feature, Tenant, PromptVersion) and add them as client defaults. A label that was not attached when the request happened cannot be sliced later.

3. Always send strings. Make it a reflex. If a header is not working, the first thing to check is its type.

4. Reproduce with curl before debugging your code. If the call works in curl, the bug is in how your program builds the request. If it fails the same way, the keys, headers or provider are at fault.

5. Keep keys in the environment. Put HELICONE_API_KEY in an environment variable, keep .env files out of git, and revoke a key immediately if it leaks.

6. Read the status code first. It tells you whether to look at your code (400), your keys (401), your limits or credit (429), or the provider (5xx).

7. Use your own request IDs. Generate a UUID, send it as Helicone-Request-Id, and keep it. Attaching feedback later becomes trivial, because you know the ID before the response arrives.

8. Name sessions by function, not by time. Paths like /research and /write produce a readable tree; paths like /step-1 and /step-2 stop meaning anything the moment the flow changes.

9. Decide on privacy before the first real user. Choose what goes into prompts, redact sensitive values in your own code, and remember that the omit headers stop storage but not transmission.

10. Check the dashboard after every change. After adding a header or a feature, send one request and look at the row. Silent failures become visible in ten seconds.

Practice cards

Habits for safe experiments

Because your traffic passes through a third party, a few habits keep experiments safe and cheap.

Use a separate Helicone API key for each project, and a separate one for experiments. A leaked experiment key can then be revoked without breaking anything else. Keys can be created and removed from the dashboard's Settings.

Test with small models and short prompts. The cheapest capable model costs a fraction of a cent for a hello-world, and the point of practice is to see the record appear, not to read long answers.

Turn on caching while you develop. A script you re-run thirty times while fixing a bug costs one provider call if the prompt does not change, and the dashboard will show you the hits as proof.

Add an Environment property from the start (local, staging, production). When the real traffic arrives, your experiments are already separable from it by one filter, and you do not have to delete anything.

Set a low rate limit on any script that loops. A while True with a bug in it can spend real money in minutes, and a policy such as "50;w=3600" turns that disaster into a 429 you can read.

Reading the dashboard without fooling yourself

A dashboard makes numbers look authoritative. Four habits keep you honest.

Treat cost as an estimate. It is computed from token counts and public prices. It is very useful for comparing features and users; for a bill, check the provider.

Look at the distribution, not only the average. A mean latency of one second can hide a tail of ten-second calls. Sort the Requests page by latency and read the slowest rows.

Filter before you conclude. A spike in cost may come from one user, one feature, or your own load test. Slicing by the labels you attached is how you find out, which is another argument for labelling early.

Remember what is missing. Requests made while logging was delayed, calls made directly to the provider, and anything you chose to omit will not be there. An absent row is not proof that nothing happened.

Choosing the integration style

A quick guide when you are starting a new project and have to pick.

Choose the gateway when you want caching, retries, rate limits and routing, and a short outage of the observability layer would be tolerable, or you are willing to add a direct-call fallback.

Choose async logging when a policy forbids putting a third party in the request path, when availability matters more than features, or when you call a model the gateway cannot reach. Use helicone-async or @helicone/async for supported providers, and the manual logger for anything else.

Whichever you choose, keep the choice in one place. Moving from async to the gateway, or away from Helicone, is cheap if your client is built in a single function and expensive if the base URL is hard-coded in forty files.

Tools worth knowing about

Three small pieces of the ecosystem help a beginner.

The MCP server @helicone/mcp lets an AI assistant query your requests and sessions in natural language. Run with npx @helicone/mcp@latest and the key in the environment. It is a convenience for exploring your own data; do not give it more access than you would give a colleague.

The export tool @helicone/export pulls your data out as JSON, JSONL or CSV, with resume and retries. Given the product's status, knowing it exists is useful even if you never use it.

HELICONE_API_KEY="$HELICONE_API_KEY" npx @helicone/export --limit 1000 --include-body

The LiteLLM route lets LiteLLM users send traffic through Helicone with a model name of the form helicone/<model>. Useful if your code already uses LiteLLM.

Things not to do

A short list of things that look reasonable and are not.

Do not put the Helicone key in front-end code. Anything in a browser bundle is public, and the key has write access to your organisation. Call the gateway from your server.

Do not use personal data as user IDs. Use an internal identifier, because these values are stored and visible in the dashboard.

Do not depend on the order in which BYOK and credits are tried. The docs describe it inconsistently, so keep your setup to one or the other.

Do not teach or rely on Experiments. The feature was removed in August 2026, although some docs pages still describe it.

Do not expose the self-hosted container to the internet with its default secrets. The defaults are for local use, and the published image predates 2026 security fixes.

Do not assume a feature header works on async logging. Caching, retries and rate limits need the gateway.

Language and platform notes

The examples use Python and Node because those are the SDKs Helicone documents most, but the gateway is plain HTTP, so any language that can set a header works. Go, Ruby, Java and C# users can call the gateway with their OpenAI-compatible client of choice, or with a raw HTTP library and the same headers.

On Windows, setting an environment variable differs by shell: $env:HELICONE_API_KEY="..." in PowerShell, set HELICONE_API_KEY=... in Command Prompt. Quoting in curl also differs: Command Prompt needs double quotes with escaped inner quotes, so PowerShell or WSL2 is easier for the examples in this guide.

On macOS and Windows, Docker runs inside a small virtual machine, so the self-hosted container's ports are published from that VM to your machine. If localhost:3000 does not load, check that Docker Desktop is running and that nothing else is using the port.

If you are behind a company network or a security gateway that inspects HTTPS, calls to the Helicone hosts may fail with certificate errors. The fix is to trust the company's root certificate in your environment, not to disable verification.

A starter setup to copy

A small module that captures most of the advice above. Save it as llm.py and import client and headers everywhere.

import os
import uuid
from openai import OpenAI

HELICONE_KEY = os.environ["HELICONE_API_KEY"]
ENVIRONMENT = os.getenv("APP_ENV", "local")

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "https://ai-gateway.helicone.ai"),
    api_key=HELICONE_KEY,
    default_headers={
        "Helicone-Property-Environment": ENVIRONMENT,
        "Helicone-Property-App": "my-app",
    },
)


def headers(user_id: str, feature: str, session_id: str | None = None,
            path: str = "/") -> dict[str, str]:
    """Per-request labels. Every value is a string."""
    h = {
        "Helicone-Request-Id": str(uuid.uuid4()),
        "Helicone-User-Id": user_id,
        "Helicone-Property-Feature": feature,
    }
    if session_id:
        h["Helicone-Session-Id"] = session_id
        h["Helicone-Session-Path"] = path
        h["Helicone-Session-Name"] = feature
    return h

Used like this:

from llm import client, headers

extra = headers(user_id="user-42", feature="refund-bot")
r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Can I return an opened item?"}],
    extra_headers=extra,
)
print(r.choices[0].message.content)
print("request id:", extra["Helicone-Request-Id"])

Notice what the design buys you. The base URL is one environment variable, so pointing away from Helicone is a configuration change. Labels are built in one place, so none can be forgotten or mistyped. Every header value is a string by construction, and the request ID is known before the call, so feedback can be attached later.

The day-one checklist

Copy this into your project notes and tick it off.

When you are stuck

Work from the outside in.

First, reproduce the failing call with curl -i. Read the status line and the response headers, especially helicone-id, Helicone-Cache and the rate-limit headers.

Second, read the error body. Helicone and the providers return readable messages, and the message names the cause more often than you would expect.

Third, find the request on the Requests page. If it is there, the gateway received it, so the problem is downstream of your code. If it is not, the request never arrived, so check the URL, the key and your network.

Fourth, check the docs page for the specific header, and check its date. If it contradicts another page, say so in your notes and trust what you observed.

Fifth, simplify. Remove every header except the authorisation, confirm a bare call works, and add headers back one at a time. The one that breaks it is your answer.

When you ask for help from a colleague or the community, bring the exact status code, the exact error body, the headers you sent with the keys removed, and what you expected to see. Those four things turn a twenty-message thread into a two-message one, and they are the same details a support engineer would ask you for.

This is part two of three. Beginner got your first requests flowing into Helicone and showed you a dashboard full of rows. At work the questions change: why did this request miss the cache, why did a provider failover happen, what happens to my app if Helicone has a bad day, and how do I prove what a feature costs per customer. This level answers those by explaining the machinery precisely enough that you can predict behaviour instead of discovering it in production.

One thing has to be said before anything else. Helicone has been in maintenance mode since it joined Mintlify on 3 March 2026. Security fixes, new models and bug fixes keep shipping, and the services remain live, but new sign-ups are disabled and the product is not growing. Much of what you read online (blog posts, tutorials, even parts of the official docs) describes a product from before that date. This guide is written against the state of things in September 2026 and flags every place where older material misleads.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a real account in a few minutes. Two assumptions apply throughout. First, that you already have a Helicone Cloud account and a HELICONE_API_KEY, because sign-ups are closed; if you do not, the self-hosting section explains what you can and cannot do locally. Second, that all header values are strings: "true", "3", never a boolean or a number. That single rule explains a surprising share of "the feature does nothing" reports.

What changed, and what it means for you

You cannot use the tool well without knowing which parts of it are current, so this section comes first. Reading the changelog-by-incident below will save you from following a tutorial into a dead end.

The integration model shifted. Older material tells you to point your SDK at oai.helicone.ai and add a Helicone-Auth header. The docs now call the provider-specific proxies (oai.helicone.ai, anthropic.helicone.ai, and the generic gateway.helicone.ai with a Helicone-Target-URL header) legacy integrations, "maintained but no longer actively developed". The recommended path is the AI Gateway at https://ai-gateway.helicone.ai, an OpenAI-compatible endpoint that takes your Helicone API key as the bearer token and translates to whichever provider serves the model. You will still maintain legacy code at work, so this guide covers both, but new code should use the gateway.

The model string format changed. The abandoned Rust gateway used base_url=".../ai" with provider/model strings such as openai/gpt-4o-mini. The current hosted gateway uses the root URL and a suffix for pinning: gpt-4o-mini/openai. Mixing the two grammars produces confusing model-not-found errors.

Pass-through billing is gated. The docs still advertise "add credits, 0% markup, no provider keys needed". Since 25 August 2026 that flow needs a ptb_enabled feature flag on your organisation, and only a small set of organisations received it. If the flag is missing, or the flag lookup fails, pass-through billing is denied. Bring-your-own-key (BYOK) is unaffected, so BYOK is the default setup to teach and to build on.

The Jawn proxy and Experiments were removed on 31 August 2026. Jawn, Helicone's backend API service, no longer proxies LLM traffic: the /v1/gateway/* routes and the OpenAI Realtime WebSocket path through Jawn are gone. The Experiments feature (endpoints, cron loop, UI) was removed too, though historical data remains in the database. If an old tutorial tells you to run experiments, it is describing something that no longer exists.

HQL got stricter on 16 September 2026. Helicone Query Language now rejects backtick or double-quoted identifiers, SET and SETTINGS, heredocs, # comments, multiple statements, database-qualified table names and ClickHouse table functions such as numbers(). Queries pasted from older posts will fail. The same fix changed how API keys created by non-owner members behave: they carry that member's user ID, so a key stops working when the member leaves the organisation. Audit who created the keys your production services use.

Self-hosted auth moved. The all-in-one image now uses Better Auth (BETTER_AUTH_SECRET, BETTER_AUTH_URL), not Supabase. Guides that mention Supabase Studio on port 8989 describe the previous architecture.

The status also changes how you design. A tool in maintenance mode is still a fine place to keep logs you already have, but you should treat adoption as reversible: keep your metadata schema portable, keep the export command handy, and keep application code free of Helicone-specific types. Later sections return to this.

The request path, precisely

The problem this section solves is prediction. To know which features work in which integration, and what breaks when something is down, you need the actual path a request takes.

There are two integration styles. In proxy or gateway integration your application sends its LLM call to Helicone, which forwards it to the provider and returns the response. Helicone is on the critical path, and you get every feature: caching, rate limits, retries, fallbacks, moderation. In async integration your application calls the provider directly and ships a log to Helicone afterwards, through OpenLLMetry (@helicone/async, helicone-async), the manual logger (@helicone/helpers, helicone-helpers) or the raw logging API. Helicone is off the critical path, and you lose everything that has to intercept a request: no cache, no rate limits, no retries.

Follow one gateway request. Your client connects to Cloudflare's network. A Cloudflare Worker, Helicone's edge code, reads your headers, applies the features you switched on (checking the cache in Workers KV, evaluating a rate-limit policy), and forwards the call to the provider. The docs state the design rule plainly: minimal business logic unless feature headers are present, and all feature code wrapped so that on failure the worker falls back to simply proxying. The response is returned to you, and only then is the log produced, placed on a Kafka queue, and consumed by Jawn. Jawn writes metadata to ClickHouse (the analytics store, main table request_response_rmt), configuration to Postgres, and the full request and response bodies to S3-compatible object storage. The dashboard is a Next.js app that reads everything through Jawn.

Three consequences follow, and each shows up in real incidents.

Logging is decoupled from the response. The log is sent after the whole response has been returned, which is why Helicone adds little latency to the answer itself. The vendor's benchmark (500 interleaved requests, an old Ada-era model) shows a mean of 2.21 s with and without Helicone. It is a vendor claim on an old model, so measure your own p95 rather than quoting it, but the architecture supports the conclusion that the hop is small.

The proxy and the logging pipeline can fail independently. In May 2026 AWS locked Helicone's account after flagging a Bedrock key as compromised (it was not). For about four days log ingestion and the dashboard were down while the proxy stayed fully functional. Logs were queued and, per the post-incident write-up, none were lost. For you that means a Helicone outage on the logging side should not break user traffic, but an outage of the edge proxy would, because it is the path your requests take.

Cache and rate-limit state live at the edge, not in your infrastructure. Cached responses sit in Cloudflare Workers KV. That matters the day a compliance team asks where prompt text is stored: for cached traffic the answer includes Cloudflare's key-value store, in Helicone's account.

The raw headers are easy to see with curl:

curl -si https://ai-gateway.helicone.ai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^helicone\|^HTTP'

With the OpenAI SDKs, client.chat.completions.with_raw_response.create(...) in Python exposes the same headers without abandoning the typed client. In Node, the .asResponse() helper on a pending call does the equivalent.

Gateway, legacy proxy and async: choosing deliberately

The problem: teams end up with all three styles in one codebase without ever choosing. Each has a different failure mode, so the choice deserves to be deliberate.

The AI Gateway is OpenAI-compatible. You set base_url to https://ai-gateway.helicone.ai, use your Helicone key as the API key, and choose any model in the registry. Endpoints are POST /chat/completions, POST /responses (the Responses API) and GET /v1/models. The gateway translates OpenAI-format requests to Anthropic, Google, Bedrock and others, which is what makes a one-line model swap possible.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.environ["HELICONE_API_KEY"],
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarise RAG in one sentence."}],
    extra_headers={
        "Helicone-User-Id": "user-123",
        "Helicone-Property-Environment": "production",
    },
)

The legacy proxies pass requests through unchanged and use a different authentication shape: the provider key goes in Authorization and the Helicone key goes in a separate Helicone-Auth: Bearer sk-helicone-... header. Existing code looks like this:

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://oai.helicone.ai/v1",
    default_headers={"Helicone-Auth": f"Bearer {os.environ['HELICONE_API_KEY']}"},
)

The generic https://gateway.helicone.ai accepts a Helicone-Target-URL header naming the real provider host (optionally with Helicone-Target-Provider), and Azure deployments add Helicone-OpenAI-Api-Base and Helicone-Model-Override. All of this is maintained but not developed, so do not build new things on it.

Async logging suits three situations: you cannot tolerate a third party in the request path, you call a provider the gateway does not serve, or you want observability only. You give up caching, rate limits and retries, but gain independence from the edge. The decision is not permanent; it is per call site, and mixing is fine if you know which path each one takes.

Provider routing and failover

The problem this solves is single-provider dependence: one provider's rate limit or outage becomes your outage. The AI Gateway moves that decision into the model string, which is a small grammar worth learning exactly.

Read the first row carefully, because it surprises people: a bare model name means cheapest first, not "the provider I configured first". If you have two BYOK providers that both serve a model, price decides, and equal prices are load-balanced. When a request must go to one provider for contractual or data-residency reasons, pin it explicitly with the suffix, and accept that a pin removes failover.

Failover triggers on 429 (rate limited), 401, 400 for context-length errors, 408, and 5xx responses. Two details matter. First, a 401 from one provider fails over to the next, which can hide a rotated or revoked key for days because traffic quietly flows elsewhere; watch for the provider mix shifting. Second, models not in the registry route only through BYOK deployments, so a fine-tuned or private model cannot benefit from automatic routing.

Routing priority is documented inconsistently. The Provider Routing page says BYOK keys are tried first and Helicone-managed keys second. The Error Handling page says that when both are configured the gateway tries pass-through billing first. Since pass-through is gated for most accounts this rarely bites, but if you have both, do not rely on the order: test it, or configure only one.

A practical pattern for regulated workloads, relevant to Gulf and Egyptian employers with data-residency rules, is to register a regional Azure OpenAI deployment as BYOK, pin the sensitive route to it with the azure/<CUID> form, and leave non-sensitive routes on automatic routing for cost. The pin keeps the prompt inside the region you chose; cache behaviour still needs separate thought, since cached bodies sit in Workers KV.

import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "https://ai-gateway.helicone.ai",
  apiKey: process.env.HELICONE_API_KEY,
});

// Sensitive route: pinned to a specific regional deployment, no failover.
const pinned = await client.chat.completions.create({
  model: "gpt-4o-mini/azure/ckx1example",
  messages: [{ role: "user", content: "Classify this ticket." }],
});

// General route: cheapest provider, automatic failover, never OpenAI.
const general = await client.chat.completions.create({
  model: "!openai,gpt-4o-mini",
  messages: [{ role: "user", content: "Draft a friendly reply." }],
});

Retries, and when they beat failover

The problem: provider APIs throw transient 429s and 5xx responses, and a naive client either crashes or hammers the provider. Helicone can retry at the edge with exponential backoff, so your application code stays simple.

Turn retries on with Helicone-Retry-Enabled: "true", and tune them with four more headers, all strings:

Retries fire on 429, 500, 502, 503 and 504, and never on other 4xx responses. That is the right behaviour: a 400 or 401 will fail identically every time, so retrying only wastes latency. Each attempt is logged as its own request, which matters for cost accounting: a request that needed three attempts shows up as three rows, and only the failed ones cost you nothing at the provider, so read the status column before you panic about duplicates.

The decision between retries and failover is simple once you see the trade. Failover tries a different provider and suits automatic routing. Retries try the same provider again and suit calls that are pinned (for residency, say) and therefore cannot fail over. The docs themselves say gateway failover is usually the better tool; use retries when you are pinned to one provider or calling through a legacy proxy.

resp = client.chat.completions.create(
    model="gpt-4o-mini/azure/ckx1example",
    messages=[{"role": "user", "content": "Classify this ticket."}],
    extra_headers={
        "Helicone-Retry-Enabled": "true",
        "Helicone-Retry-Num": "3",          # a string, not an int
        "Helicone-Retry-Factor": "2",
        "Helicone-Retry-Max-Timeout": "8000",
    },
)

Numeric header values are silently ignored, which is why "my retry settings do nothing" is the single most common report. If you use the OpenAI Python SDK, extra_headers={"Helicone-Retry-Num": 3} will send an integer where a string is expected; write "3".

The response cache, precisely

The problem this solves is paying twice for the same answer. The cache is powerful, but it disappoints people, and every disappointment traces to how the cache key is built.

Enable it with Helicone-Cache-Enabled: "true". The key is a hash of: the seed, the full URL, the entire request body, relevant headers including Authorization, and the bucket index. Storage is Cloudflare Workers KV at the edge. Because the whole body is hashed, anything in it that varies between calls produces a different key and a miss: a timestamp in the system prompt, a per-request ID, a slightly different temperature, reordered JSON keys from a serialiser, or an extra message.

Buckets deserve an explanation, because they change what "cache hit" means. With the default size of 1, the first answer is stored and every later identical request replays it. With Helicone-Cache-Bucket-Max-Size: "3", the first three identical requests each generate and store a fresh answer; later ones return one of the three at random. You trade determinism for variety, which is useful for chatbots where users would notice identical replies, and wrong for anything that must be reproducible.

Seeds isolate tenants. If two customers might send byte-identical prompts but must never see each other's answers (because an answer depends on hidden context, say), set Helicone-Cache-Seed to the customer or user ID. Since Authorization is also in the key, different Helicone keys never share entries anyway, but seeds let you separate within one key.

Ignore keys fix the most common miss. If your client adds request_id or timestamp to the body, list them: Helicone-Cache-Ignore-Keys: "request_id,timestamp". The same header is recommended alongside provider prompt caching, where you would ignore prompt_cache_key so it does not fragment the cache.

resp = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain idempotency in one line."}],
    extra_headers={
        "Helicone-Cache-Enabled": "true",
        "Cache-Control": "max-age=3600",
        "Helicone-Cache-Seed": "tenant-42",
        "Helicone-Cache-Ignore-Keys": "request_id",
    },
)
print(resp.headers.get("Helicone-Cache"))          # HIT or MISS
print(resp.headers.get("Helicone-Cache-Bucket-Idx"))

When not to cache: anything personalised beyond what the body shows, anything time-sensitive ("what is the latest..."), anything where a stale answer is worse than a slow one, and non-deterministic generation where users expect variety. Caching also interacts with privacy: a cached body lives in Workers KV for the TTL, and Helicone-Omit-Request and Helicone-Omit-Response only stop storage in your logs; the bodies still transit Helicone's backend.

To measure whether caching pays, filter the Requests page on cache hits and compare cost and latency. A hit rate under about ten percent on a route usually means the body is not stable enough, not that the route is uncacheable: start with the ignore keys.

Rate limits as a product feature

The problem: one abusive user, one runaway loop or one expensive feature can burn your whole budget before any alert fires. Helicone enforces limits at the edge, before the provider is called, using a single header.

The policy grammar is [quota];w=[seconds];u=[request|cents];s=[user|<property>], following the IETF rate-limit header draft with a custom s for segmentation:

The window w must be at least 60 seconds. Unit u defaults to request, and cents makes it a spend limit, which is the more useful form for LLM apps because a single request's cost varies by two orders of magnitude. Exceeding a limit returns 429, with Helicone-RateLimit-Limit, Helicone-RateLimit-Remaining and Helicone-RateLimit-Policy on the response. Token-based limits and multiple policies per request are documented as "coming soon", so you cannot express "both per-user and global" in one call today.

The trap that makes limits look broken is segmentation without the segment: s=user needs Helicone-User-Id on the request, and s=organization needs Helicone-Property-Organization. If the header is missing, the limit simply does not apply per user, and you find out when the bill arrives.

def ask(user_id: str, tenant: str, prompt: str):
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        extra_headers={
            "Helicone-User-Id": user_id,
            "Helicone-Property-Organization": tenant,
            "Helicone-RateLimit-Policy": "500;w=3600;u=cents;s=user",
        },
    )

Your application needs to treat a 429 from the gateway correctly. A rate-limit 429 is a business decision, not a transient fault: do not retry it in a tight loop, and do not let Helicone's own retries (which also trigger on 429) fight it. The response headers tell you when budget returns; surface a friendly message instead.

There is a real design question about where to enforce. Edge limits are cheap and block spend early, but they are keyed only on what you put in headers, and the limit state lives in Workers KV, which is eventually consistent. Treat them as a guardrail against runaway cost, not as a precise ledger. If billing accuracy matters, reconcile from the logged costs (see HQL below), and keep authoritative quota checks in your own service.

Metadata design: users, properties and sessions

The problem: a dashboard is only as good as the labels on the data. Teams that add properties ad hoc end up with env, Env, environment and Environment as four separate columns, and nobody can answer "what did the summarise feature cost in EU production last week".

Three mechanisms carry metadata. Helicone-User-Id identifies the end user and powers user metrics and per-user rate limits. Helicone-Property-<Name> adds arbitrary key-value pairs, with no restriction on values. Helicone-Session-* groups requests into a run. The discipline is to decide the schema once, centrally, and enforce it in a wrapper.

A workable starting schema:

Keep cardinality in check. Tenant with a few hundred values is fine and useful; a property holding a per-request ID is a log field pretending to be a dimension, and it makes every group-by unreadable. Keep personal data out of properties and user IDs too: use an opaque internal ID, not an email address, because these values appear in the dashboard, webhooks and exports.

Put the schema in code so nobody types header names by hand:

from dataclasses import dataclass

@dataclass(frozen=True)
class CallContext:
    user_id: str
    tenant: str
    feature: str
    route: str
    environment: str = "production"
    version: str = "unknown"

    def headers(self) -> dict[str, str]:
        return {
            "Helicone-User-Id": self.user_id,
            "Helicone-Property-Environment": self.environment,
            "Helicone-Property-Feature": self.feature,
            "Helicone-Property-Tenant": self.tenant,
            "Helicone-Property-Route": self.route,
            "Helicone-Property-Version": self.version,
        }

def chat(ctx: CallContext, **kwargs):
    return client.chat.completions.create(extra_headers=ctx.headers(), **kwargs)

You can also set or correct a property after the fact: PUT /v1/request/{requestId}/property on https://api.helicone.ai (EU: https://eu.api.helicone.ai) with a body such as {"Environment": "production"}, authenticated with your Helicone key as a bearer token. The request ID is the helicone-id response header, or an ID you pre-set with Helicone-Request-Id.

Sessions and traces

A session groups requests that belong together, such as an agent run or a conversation. Three headers define it: Helicone-Session-Id (a UUID you generate once per run), Helicone-Session-Path (a slash-separated location in the trace tree, for example /task/research), and Helicone-Session-Name (the session type, such as Customer Support, so runs of the same kind can be compared). The docs treat all three as required for the trace view to work.

The path is where people go wrong. It builds the parent-child tree, so /plan and /plan/search make the second a child of the first. Design paths by function, not by time: /task/research/fetch describes what the step is, while /step-3 tells you nothing once the agent's plan changes. Start every path with /.

import uuid

session_id = str(uuid.uuid4())

def step(path: str, **kwargs):
    return client.chat.completions.create(
        extra_headers={
            "Helicone-Session-Id": session_id,
            "Helicone-Session-Path": path,
            "Helicone-Session-Name": "Research Agent",
        },
        **kwargs,
    )

plan = step("/plan", model="gpt-4o-mini",
            messages=[{"role": "user", "content": "Plan a market scan."}])
found = step("/plan/search", model="gpt-4o-mini",
             messages=[{"role": "user", "content": "Run step one."}])

Tool calls and vector-database lookups are not LLM requests, but you can log them into the same session with the logger SDK or cURL so the trace shows the whole run, not only the model calls. That is what turns a pile of rows into a debugging tool: you open one session and see that the slow step was a retrieval, not the model.

Versioned prompts through the gateway

The problem: prompts edited inside application code change with every deploy, cannot be rolled back independently, and are invisible to the people who tune them. Helicone's prompt management stores templates centrally, versions them, and lets a request reference one by ID.

A prompt has a six-character alphanumeric ID, versions, and environments (production, staging, development, or custom ones), so "what is live" is an environment pointer rather than a deploy. Templates contain variables written {{hc:name:type}} and partials written {{hcp:prompt_id:index:env}} that embed another prompt's message. Partials resolve before variables, so a partial may itself contain variables that then get filled.

The current system replaces the older Helicone-Prompt-Id header with hpf and hpstr helper functions. That legacy approach and the legacy editor pages are still documented, which causes confusion; the current one sends prompt_id and inputs in the request body through the AI Gateway:

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Keep it under 80 words."}],
    extra_body={                        # fields the SDK does not know about
        "prompt_id": "abc123",
        "environment": "production",    # default is production
        "inputs": {"customer_name": "Noor", "ticket_count": 3},
    },
)

prompt_id, inputs and model are required. environment defaults to production. Any messages you add are appended to the saved prompt, which is how you pass the live user turn while the system instructions stay in Helicone. The OpenAI Python SDK rejects unknown keyword arguments in strict typing, so pass them through extra_body={...}; in TypeScript, cast the parameters with HeliconeChatCreateParams from @helicone/helpers.

Variable types deserve precision. The type is string, number, boolean or anything else you care to name, but only number and boolean are actually type-checked. Booleans accept true, false, "true", "false", "yes" and "no". When a variable is the entire value of a JSON string field, it is replaced by the typed value rather than a string, so an array can land in an enum. Partials are available for Chat Completions on the AI Gateway.

If you want the prompt body without routing the call through the gateway (for example because you use async logging), the SDK fetches and assembles it:

import { HeliconePromptManager } from "@helicone/helpers";
import OpenAI from "openai";

const pm = new HeliconePromptManager({ apiKey: process.env.HELICONE_API_KEY });
const openai = new OpenAI();

const { body, errors } = await pm.getPromptBody({
  prompt_id: "abc123",
  inputs: { customer_name: "Noor" },
});
if (errors?.length) throw new Error(JSON.stringify(errors));
const resp = await openai.chat.completions.create(body);

Python offers HeliconePromptManager(api_key=...).get_prompt_body({...}) from helicone_helpers. Check errors every time: a missing variable or a type mismatch is reported there rather than thrown, and sending a half-filled prompt to a model is worse than failing.

Operationally, treat prompts like code. Use environments so staging can test a new version while production stays put, promote by moving the environment, and record which prompt version served a request. Keep a copy of important prompts in your own repository as well: with the product in maintenance mode, a prompt that lives only in a vendor's database is a migration risk, and the export tooling covers request logs far better than it covers template history.

Streaming cost, scores and feedback

The problem: a dashboard cost figure you cannot trust is worse than none. Two common causes are streaming and evaluation data.

Streaming. Providers report token usage in a final chunk that many clients never ask for. Without it, Helicone cannot compute cost correctly, and you see zero or wrong figures for streamed calls. The fix is stream_options={"include_usage": True} on the request, or the header helicone-stream-usage: "true" when your client cannot set the option. Do this for every streaming call path, then check one streamed request's cost against the provider's own usage page.

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about latency."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

For a model whose name Helicone cannot resolve from the URL or body, such as a custom or Azure deployment name, cost comes out missing; set Helicone-Model-Override so the cost registry can price it.

Feedback and scores. Both attach judgements to a request after the fact, using the REST API at https://api.helicone.ai (EU: https://eu.api.helicone.ai) with Authorization: Bearer <HELICONE_API_KEY>.

Scores must be integers or booleans. A float like 0.92 is rejected, so scale to an integer (92) and document the scale. Scores take about ten minutes to aggregate in dashboards, so do not build a real-time UI on them.

import os, requests

API = "https://api.helicone.ai"   # EU: https://eu.api.helicone.ai
H = {"Authorization": f"Bearer {os.environ['HELICONE_API_KEY']}"}

def record_thumb(request_id: str, up: bool):
    r = requests.post(f"{API}/v1/request/{request_id}/feedback",
                      headers=H, json={"rating": up}, timeout=10)
    r.raise_for_status()

def record_scores(request_id: str, relevance_pct: int, grounded: bool):
    r = requests.post(f"{API}/v1/request/{request_id}/score",
                      headers=H,
                      json={"scores": {"relevance": relevance_pct, "grounded": grounded}},
                      timeout=10)
    r.raise_for_status()

The practical question is how to get the request ID back to the place where the feedback happens. Either capture helicone-id from the response and return it to the browser with the answer, or pre-set Helicone-Request-Id to a UUID you already store against the message. The second is more robust, because it survives retries of your own handler and lets a background evaluator score a request it never saw the response headers for.

This is where Helicone meets the neighbouring tools in this catalogue. If you run an offline evaluation with RAGAS, post its metrics as integer scores on the logged requests, so production traffic and evaluation live on the same rows. If you are comparing tracing tools, Langfuse covers a similar observability space with a different data model, and Helicone itself documents a Langfuse integration.

Querying your data: REST, HQL and the MCP server

The problem: the dashboard answers the questions its authors anticipated. Anything else, such as "every request from tenant 42 that cost more than a cent last Tuesday", needs the query surfaces.

The request query API. POST https://api.helicone.ai/v1/request/query-clickhouse takes {"filter": {...}, "limit": 100, "offset": 0} and is the one optimised for bulk reads. For a point lookup, POST /v1/request/query exists but can time out over large ranges. The most famous trap: property filters must be wrapped in the table name.

curl -s https://api.helicone.ai/v1/request/query-clickhouse \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "request_response_rmt": {
        "properties": { "Environment": { "equals": "production" } }
      }
    },
    "limit": 100,
    "offset": 0
  }'

Without the request_response_rmt wrapper the call succeeds and returns [], which reads as "no data" rather than "bad filter". Combine filters with {"left": ..., "operator": "and" | "or", "right": ...}. Related endpoints are GET /v1/request/{id}, POST /v1/session/query and POST /v1/user/query.

HQL, Helicone Query Language. This is read-only ClickHouse SQL over request_response_rmt, executed with POST /v1/helicone-sql/execute and a body {"sql": "..."}. GET /v1/helicone-sql/schema lists the columns, POST /v1/helicone-sql/download returns a CSV through a signed URL, and saved queries live under /v1/helicone-sql/saved-queries. It is available only on selected workspaces (by request) and on the Pro plan and above, so check your access before designing around it. Limits are 300,000 rows, a 30-second timeout, 100 queries per minute and 10 CSV downloads per minute.

Two details trip up everyone. Cost columns are stored multiplied by 1,000,000,000, so divide by 1e9 to get dollars. And since the September 2026 hardening, only plain identifiers are allowed and the FROM target must be request_response_rmt or a parenthesised subquery.

SELECT
  properties['Tenant']                AS tenant,
  request_model,
  count()                             AS calls,
  sum(cost) / 1000000000.0            AS cost_usd,
  avg(total_tokens)                   AS avg_tokens
FROM request_response_rmt
WHERE request_created_at >= now() - INTERVAL 7 DAY
  AND status = 200
GROUP BY tenant, request_model
ORDER BY cost_usd DESC
LIMIT 50

Notice what the query avoids: no backticks, no SETTINGS, no database prefix. If you paste a query from a blog and get a validation error, rewrite it to plain identifiers first.

The MCP server lets an AI assistant query your logs directly. Add it to your client configuration:

{
  "mcpServers": {
    "helicone": {
      "command": "npx",
      "args": ["@helicone/mcp@latest"],
      "env": { "HELICONE_API_KEY": "sk-helicone-REPLACE_ME" }
    }
  }
}

Its tools are query_requests (filter, offset, limit defaulting to 100, sort, and includeBodies) and query_sessions (which requires startTimeUnixMs, endTimeUnixMs and timezoneDifference). Keep includeBodies off unless you need it: bodies can hold personal data, and pulling them into an assistant's context is a data-handling decision. Also note the npm package is at version 0.1.6 and was last published in November 2025, so pin it (@helicone/mcp@0.1.6) rather than tracking latest in anything shared.

Async logging: Helicone off the critical path

The problem: some teams cannot accept any third party in the request path, or call models the gateway does not serve. Async integration keeps the provider call direct and ships telemetry separately. You give up caching, rate limits, retries and bucket caching, and gain independence: if Helicone is unreachable, your calls are unaffected.

helicone-async (Python) and @helicone/async (Node) build on OpenLLMetry instrumentation. You initialise once, and supported SDK calls are captured automatically. Supported providers are OpenAI, Anthropic, Azure OpenAI, Cohere, Bedrock and Google AI Platform.

import os
from helicone_async import HeliconeAsyncLogger

logger = HeliconeAsyncLogger(api_key=os.environ["HELICONE_API_KEY"])
logger.init()

logger.set_properties({
    "Helicone-Session-Id": "8c1f2a5e-0d44-4c26-9d0a-3f5bc1a0e001",
    "Helicone-Session-Path": "/abstract",
    "Helicone-Session-Name": "Course Plan",
})

# Privacy switches
# logger.disable_content_tracing()   # send metrics only, no bodies
# logger.disable_logging()           # send nothing at all (async mode only)

In Node the shape is new HeliconeAsyncLogger({ apiKey, providers: { openAI: OpenAI } }).init(), with logger.withProperties({...}, fn) to scope properties to a block. The two privacy switches matter: disable_content_tracing keeps token counts, latency and cost while withholding prompt and completion text, which is often the compromise a security review wants.

For a model with no automatic instrumentation, the manual logger records a call you made yourself:

from helicone_helpers import HeliconeManualLogger

logger = HeliconeManualLogger(
    api_key=os.environ["HELICONE_API_KEY"],
    headers={"Helicone-Property-Feature": "triage"},
)

def call_model(recorder):
    result = my_private_model.generate(prompt)          # your own client
    recorder.append_results({"text": result.text})
    return result

request = {"model": "my-private-model", "messages": [{"role": "user", "content": prompt}]}
logger.log_request(request=request, operation=call_model, provider="custom")

log_request times your operation, and send_log is the lower-level form taking a provider, request, response and a LoggingOptions with start_time, end_time, additional_headers and time_to_first_token_ms. The default logging endpoint is https://api.worker.helicone.ai. Go, TypeScript and cURL variants exist.

The hybrid pattern is underrated: route the bulk of traffic through the gateway for caching and failover, and use async or the manual logger for the one path that must stay direct. Both feed the same dashboard, distinguished by a property such as Transport.

Webhooks and alerts: pushing instead of polling

The problem: dashboards only help when someone looks. Two features push information out: alerts notify people, and webhooks deliver each request to your own code.

Alerts live under Settings, Alerts. You choose a metric (error rate, cost, latency, total, prompt or completion tokens, prompt-cache reads and writes, or count), a threshold, a time frame and a minimum number of requests. You can group by user, model, provider or a custom property, apply an aggregation (sum, average, minimum, maximum or a percentile), and filter. Notification channels are email and Slack.

Two settings separate a useful alert from a noisy one. The minimum request count prevents a 50% error rate from firing on two calls at 3 a.m. The grouping lets one alert watch every tenant: "cost per Tenant above $50 in an hour" is a single rule that catches the one runaway customer. Start with three: error rate over 5% with a minimum of 50 requests, hourly cost per tenant, and p95 latency for your most important route. Add more only when an incident shows the gap.

Webhooks send a signed HTTP POST per logged request to a URL you own. Configure them in the dashboard under Webhooks or with POST /v1/webhooks (plus GET and DELETE). Settings are the destination URL, a sample rate from 0 to 100 (default 100), whether to include data (default on), and property filters that must all match. Since March 2025 the payload carries a unified request_response_url, a signed S3 URL pointing to the full request and response.

Facts to design around: bodies embedded in the payload are truncated at 10 KB; the request_response_url expires after 30 minutes, so fetch it promptly or not at all; and delivery times out after 2 minutes, so acknowledge fast and do real work asynchronously. Always verify the signature, because the endpoint is public by necessity. The helicone-signature header is an HMAC-SHA256 of JSON.stringify(body) with your webhook secret, and the comparison must be timing-safe.

import crypto from "node:crypto";
import express from "express";

const app = express();
app.use(express.json({ limit: "2mb" }));

app.post("/helicone-webhook", (req, res) => {
  const signature = req.header("helicone-signature") ?? "";
  const expected = crypto
    .createHmac("sha256", process.env.HELICONE_WEBHOOK_SECRET!)
    .update(JSON.stringify(req.body))
    .digest("hex");

  const a = Buffer.from(signature);
  const b = Buffer.from(expected);
  if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
    return res.status(401).end();
  }

  queue.enqueue(req.body);      // your own queue: acknowledge first, process later
  res.status(200).end();
});

Hashing a re-serialised body works only if the key order survives the round trip, which is why some teams capture the raw bytes instead; test with a real delivery before trusting the check in production. The exact digest encoding (hex versus base64) should be confirmed against the current webhook docs when you implement it.

Good uses for webhooks: copying selected requests (filtered by property) into your own warehouse for long-term retention, feeding an evaluation pipeline, and triggering a review workflow when a score is low. Sampling at, say, 10% keeps the volume manageable when you only need a statistical view.

Guardrail headers: moderation, security and token limits

The problem: some failures are caused by input, not by the model. Helicone offers three header-switched guards. They are useful, limited, and narrower than their names suggest.

Helicone-Moderations-Enabled: "true" calls OpenAI's moderation endpoint using your OpenAI key configured in Helicone, on the latest user message. It works for OpenAI only. A flagged prompt returns HTTP 400 with code PROMPT_FLAGGED_FOR_MODERATION and the message "The given prompt was flagged by the OpenAI Moderation endpoint."

Helicone-LLM-Security-Enabled: "true" screens with Meta's Prompt Guard (86M) for jailbreaks and direct or indirect prompt injection in eight languages. Adding Helicone-LLM-Security-Advanced: "true" also runs Llama Guard (3.8B) across fourteen categories. It supports OpenAI models only. A blocked call returns {"success": false, "error": {"code": "PROMPT_THREAT_DETECTED", ...}} with the message "Prompt threat detected. Your request cannot be processed."

Helicone-Token-Limit-Exception-Handler takes truncate, middle-out or fallback to cope with prompts that exceed the context window. With fallback, list two models in the body ("gpt-4o, gpt-4o-mini"; only the first two are used). Token counting is an estimate, and reserved output (max_tokens) counts against the limit, so leave headroom. If the body has no model, supply Helicone-Model-Override.

Keep your expectations realistic. These are edge filters, and a model-based classifier has false negatives and false positives. They do not replace input validation, output checks, least-privilege tool access and an allow-list of actions in your application. For a fuller treatment of validating model inputs and outputs in code, read the Guardrails guide. In your app, handle PROMPT_FLAGGED_FOR_MODERATION and PROMPT_THREAT_DETECTED as first-class outcomes with a designed user message, not as generic 500s.

Self-hosting: what is realistic

The problem: for learners without a cloud account, and for teams with strict residency rules, self-hosting is attractive. It is possible and useful, but with limits that the docs do not spell out well.

The supported quick path is the all-in-one Docker image helicone/helicone-all-in-one (the newest tag is v2025.08.21, identical to latest). It bundles the dashboard, Jawn, Postgres, ClickHouse and MinIO:

docker run -d --name helicone \
  -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -v helicone-postgres:/var/lib/postgresql/data \
  -v helicone-clickhouse:/var/lib/clickhouse \
  -v helicone-minio:/data \
  helicone/helicone-all-in-one:v2025.08.21

curl http://localhost:8585/healthcheck

Open http://localhost:3000. The three volumes matter: without them a container restart wipes everything. For any non-local deployment, every URL variable must share one origin, and browsers must be able to reach all three ports, because the dashboard calls Jawn and MinIO from the user's browser, not from the server:

export PUBLIC_URL="https://helicone.internal.example.com"
docker run -d --name helicone -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -e SITE_URL="$PUBLIC_URL" \
  -e BETTER_AUTH_URL="$PUBLIC_URL" \
  -e BETTER_AUTH_SECRET="$(openssl rand -base64 32)" \
  -e NEXT_PUBLIC_APP_URL="$PUBLIC_URL" \
  -e NEXT_PUBLIC_HELICONE_JAWN_SERVICE="https://helicone-api.internal.example.com" \
  -e NEXT_PUBLIC_IS_ON_PREM=true \
  -e S3_ENDPOINT="https://helicone-s3.internal.example.com" \
  -e S3_ACCESS_KEY="$MINIO_KEY" -e S3_SECRET_KEY="$MINIO_SECRET" \
  helicone/helicone-all-in-one:v2025.08.21

The failure modes map directly to variables, which makes debugging mechanical:

There is no email service in the container, so a new user's email must be verified by hand (UPDATE "user" SET "emailVerified" = true WHERE email = ... through docker exec -u postgres helicone psql -d helicone_test), and a user with no organisation needs one created and joined to.

Three caveats deserve to be said plainly. First, the all-in-one image runs the dashboard and Jawn only: since Jawn's proxy was removed, LLM traffic cannot be proxied through it, and the docs' pointer to a self-hosted AI Gateway leads to a page that no longer exists while the standalone Rust gateway is abandoned. For self-hosting, log through the manual logger or async path. Whether the manual logger's logging_endpoint works against your own Jawn is not documented, so prove it in a lab before you rely on it. Second, the image predates all the 2026 security fixes (the HQL cross-tenant bypass, the admin-takeover path, provider-key leakage between organisations), so keep it off the public internet and treat it as an internal, single-team tool, or build from main. Third, there are no maintained release tags after August 2025 and no published upgrade guide; "upgrade" means pin, back up Postgres, ClickHouse and MinIO, test, then roll. If you build from main you must also apply the Postgres migrations, including running the audit query in the proxy-key migration header before VALIDATE CONSTRAINT.

Sizing guidance from the docs is 4 vCPU, 16 GB RAM and 100 GB of disk for a single node. Put TLS in front with a reverse proxy, change the default BETTER_AUTH_SECRET (change-me-in-production) and the MinIO minioadmin credentials, and keep Postgres (5432) and ClickHouse (8123) unreachable from outside the host. For Kubernetes there are Helm charts (helicone-core, helicone-infrastructure, helicone-monitoring, helicone-argocd) in the helicone-helm-v3 repository, installed per chart with helm upgrade --install after copying the example values and secrets files. See the Helm and Kubernetes guides for the mechanics, and Docker for the volume and networking behaviour behind the table above.

Configuration, secrets and CI

The problem: Helicone settings end up scattered across environment variables, headers in code and dashboard toggles, and nobody can say which one wins in production.

Organise them in layers. Secrets (HELICONE_API_KEY, any webhook secret) come from the environment or a secret manager and never from source control; the key format is sk-helicone-..., and on Windows PowerShell you set it with $env:HELICONE_API_KEY="sk-helicone-...". Provider keys are not in your code at all when you use the gateway with BYOK: they live in Helicone's Provider Settings, so rotation happens there, and the place to notice a revoked key is your provider mix, not your app logs. Behaviour (caching, retries, rate limit policies) is expressed as headers, and you should centralise it in one module so the policy per route is reviewable in a pull request.

# llm_policy.py: one place where gateway behaviour is decided
POLICIES = {
    "summarise": {
        "Helicone-Cache-Enabled": "true",
        "Cache-Control": "max-age=86400",
        "Helicone-Retry-Enabled": "true",
        "Helicone-Retry-Num": "2",
    },
    "chat": {
        "Helicone-RateLimit-Policy": "300;w=3600;u=cents;s=user",
    },
    "residency-pinned": {
        "Helicone-Retry-Enabled": "true",
        "Helicone-Retry-Num": "3",
    },
}

def policy(name: str) -> dict[str, str]:
    return dict(POLICIES[name])       # copy, so callers cannot mutate shared state

A key that is shared across every service makes attribution and revocation hard. Create a key per service or environment where the plan allows, name it clearly, and remember the 16 September 2026 behaviour: keys created by a non-owner member carry that member's user ID and stop working when they leave. Have an owner or a service identity create production keys.

In CI, never call the real gateway from unit tests. Wrap your client behind a function, substitute a fake that returns canned responses, and assert on the headers you would have sent. That tests policy (did summarise turn caching on, did the tenant property get set) without spending money or depending on the network:

def test_summarise_sets_cache_and_tenant(fake_client):
    ctx = CallContext(user_id="u1", tenant="t9", feature="summarise", route="/summarise")
    chat(ctx, model="gpt-4o-mini", messages=[{"role": "user", "content": "hi"}])
    sent = fake_client.last_headers
    assert sent["Helicone-Property-Tenant"] == "t9"
    assert sent["Helicone-Cache-Enabled"] == "true"
    assert all(isinstance(v, str) for v in sent.values())   # the string rule

The last assertion encodes the most valuable single check. For an integration test, run a small number of calls against a staging key with Environment=staging, and assert on a Helicone-Cache: HIT on the second call rather than on model text. Keep these in a separate, opt-in job, because they cost money and depend on an external service. The GitHub Actions guide shows how to gate such a job behind a label or a schedule, and Promptfoo covers prompt regression testing, which is where checking the quality of answers belongs.

Multi-tenant attribution and cost control

The problem: a product with many customers needs to know who costs what, and stop one from harming the others. The pieces already exist; this section shows how they fit.

Put the customer in Helicone-Property-Tenant and, where individual end users matter, in Helicone-User-Id. From that you get per-tenant spend on the dashboard, per-tenant alerts (group by Tenant), per-tenant rate limits (s=Tenant with the matching property), per-tenant webhook filters, and per-tenant cache isolation with Helicone-Cache-Seed. Helicone organisations are a different thing, your own account structure (one on Hobby, five on Team, unlimited on Enterprise), and are not the right unit for modelling your customers.

Cost is computed from the provider's reported usage and Helicone's open-source cost registry (the docs mention more than 300 models). Two practical habits follow. Reconcile monthly: export the sum of logged cost and compare it with the provider invoice; a persistent gap points at missing stream usage, unresolved model names, or calls that bypass Helicone altogether. And set layered limits: a cost-based per-user limit at the edge for runaway loops, and a per-tenant alert for budgets.

Cost-saving levers, in order of leverage: the response cache on stable routes, routing to the cheapest capable model (the automatic model string does this across providers for the same model, while choosing a cheaper model is a product decision), trimming prompts (look at prompt token averages by feature), and cost-based limits. Retry storms also cost money, so watch the error rate next to spend.

Pricing and plans are worth knowing as history rather than as a purchase path: the pricing page still lists Hobby (free, 10,000 requests a month), Pro ($79 a month), Team ($799 a month) and Enterprise, with retention of 7 days, 1 month, 3 months and configurable respectively, but self-serve upgrades have been removed. Plan retention matters operationally, because old logs age out: if you need a year of history, copy it elsewhere on a schedule (webhooks or export) instead of assuming it will be there.

Exporting data and planning an exit

The problem: with the product in maintenance mode, the responsible position is that you can leave whenever you choose. Vendors in this position often stay up for a long time, but you should not need to find out the hard way how to get your data.

Helicone ships an export CLI, @helicone/export, version 1.0.0 as of October 2025. It supports checkpoint and resume, retries with backoff, and JSON, JSONL or CSV output:

HELICONE_API_KEY="sk-helicone-REPLACE_ME" npx @helicone/export \
  --start-date 2026-01-01 --end-date 2026-01-31 \
  --property Environment=production \
  --format jsonl --include-body

# EU region
HELICONE_API_KEY="sk-helicone-REPLACE_ME" npx @helicone/export --region eu --limit 10000 --include-body

Data export is tied to paid plans, and retention limits what exists to be exported, so schedule it rather than running it once in a panic. Decide what you need: bodies are the bulky, sensitive part, while metadata (model, tokens, cost, latency, properties, sessions) is what dashboards and audits need. Store exports encrypted and apply the same retention policy you would apply to the original logs.

Keep the rest of the system exit-friendly. Hold your metadata schema in your own code, as shown earlier. Keep prompts in version control as well as in Helicone. Isolate the client behind a function so that swapping the base URL is a one-file change, since the gateway is OpenAI-compatible and the same pattern works with other OpenAI-compatible gateways. Practitioners name Langfuse, the LiteLLM proxy, OpenRouter and Portkey as common destinations; that is a description of what people do, not an endorsement. If observability is the main need, the Langfuse guide shows a destination built around traces and evaluations.

A migration rehearsal is cheap: export a week, load it into a scratch store, and check that you can still answer your three most important cost questions. If you can, your exit is real.

Debugging beyond the dashboard

The problem: "it did not work" arrives without context. A fixed order of checks turns that into a five-minute diagnosis.

Start with the status and headers of the failing call. A 401 on the AI Gateway means the wrong key type: only the Helicone key belongs in Authorization, and a BYOK provider error returns "type": "invalid_api_key" with "Authentication failed". A 429 with "Insufficient credits" means you are on pass-through billing without credits or without the gated flag, and the fix is to add provider keys in Provider Settings. A 429 with Helicone-RateLimit-Remaining: 0 is your own rate-limit policy. A 400 with PROMPT_FLAGGED_FOR_MODERATION or PROMPT_THREAT_DETECTED is a guard. A 403 about a suspended wallet or blocked model is account-level and goes to support.

Next, "the feature does nothing". Nine times out of ten the cause is one of four things: a header value that is not a string, a segmentation header missing (s=user without Helicone-User-Id), a feature used through the wrong integration style (caching with async logging), or a body that varies between calls (cache misses).

Third, "the numbers are wrong". Zero cost on a streamed request means no usage chunk (include_usage); missing cost on a custom model means Helicone-Model-Override; an empty query result means a filter without the request_response_rmt wrapper.

For self-hosted installs the checks are environmental: the Jawn health check at /healthcheck, the __ENV.js file the browser actually loads, and one consistent origin across the URL variables. One specific known fault: a 400 "Invalid token" on the dashboard with a log line "Node.js 20 detected without native WebSocket support" comes from a Supabase realtime client on Node 20, fixed in May 2026, so it affects the August 2025 image and is cured by building a newer main.

Putting it all together

Assemble everything into one small, realistic service: a support-ticket assistant for a multi-tenant product.

1. One client module. A single llm.py builds the gateway client with base_url="https://ai-gateway.helicone.ai" and the Helicone key from the environment. Application code never constructs an OpenAI client directly.
2. A context object. CallContext produces Helicone-User-Id and the Environment, Feature, Tenant, Route and Version properties. Every call needs one.
3. Named policies. summarise turns on a 24-hour cache with two retries. chat adds a cost-based per-user limit of $3 an hour. residency-pinned pins to a regional BYOK deployment with retries and no failover. The policy module is reviewed like any other code.
4. Sessions for agent runs. The triage agent generates a session ID per ticket, paths like /triage/classify and /triage/draft, and a session name of Ticket Triage.
5. Prompts in Helicone and in git. The classification prompt lives in Helicone with staging and production environments, and is mirrored in the repository.
6. Feedback loop. The UI returns the helicone-id with each draft; agents' thumbs go to the feedback endpoint, and a nightly RAGAS-style evaluation posts integer scores.
7. Guards handled. Moderation and security errors have designed responses; screened routes are pinned to OpenAI.
8. Alerts and reconciliation. Error rate, hourly cost per tenant and p95 latency alerts notify Slack; a monthly job compares logged cost with the provider invoice.
9. Exit readiness. A weekly export job writes metadata JSONL to your own storage, and the client is one file away from a different base URL.
10. Tests. Unit tests assert the headers with a fake client; one opt-in job proves a cache hit against staging.

ctx = CallContext(user_id="u-8841", tenant="acme-eg", feature="summarise",
                  route="/tickets/summary", environment="production", version="a1b2c3d")

resp = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": ticket_text}],
    extra_headers={**ctx.headers(), **policy("summarise")},
)
request_id = resp.headers["helicone-id"]
summary = resp.parse().choices[0].message.content

The value is in the combination. Properties make the cost legible, policies make behaviour reviewable, the request ID closes the loop with feedback, and the export keeps the decision reversible.

Where you are now

You can place any Helicone call on the map: gateway, legacy proxy or async, and you know what each one gives and withholds. You can route with the model-string grammar, pin when a contract requires it, and explain why failover can hide a dead key. You know how the cache key is built, so you can predict a miss, and you can enforce spend with edge rate limits. You can design a metadata schema, trace agent runs with sessions, version prompts by environment, record scores and feedback correctly, query through REST and HQL, and push events out through alerts and webhooks. You know what self-hosting can and cannot do today, and you have an exit plan.

The next level, senior, takes this from "a tool I use well" to "a platform I am accountable for": the failure modes of the edge and the logging pipeline, security and tenancy in depth, governance, and the honest question of when to migrate. Before you go, revisit the Docker and Kubernetes guides if the self-hosting section felt thin, and Langfuse if you want to compare approaches.

Sources

• Helicone documentation index (llms.txt)
• Quick start
• AI Gateway overview
• Provider routing
• Error handling and fallback
• Prompt integration
• Header directory
• Caching
• Custom rate limits
• Retries
• Custom properties
• Scores
• Feedback
• LLM security
• Moderations
• Token limit exception handlers
• Prompts overview
• Prompts SDK
• Sessions
• HQL
• Webhooks
• Alerts
• OpenLLMetry async integration
• Manual logger (Python)
• MCP server
• Query requests (ClickHouse) API
• Self-hosting overview
• Self-host with Docker
• Self-host on Kubernetes
• Proxy versus async
• How Helicone calculates cost
• Enable stream usage
• Helicone is joining Mintlify
• AWS account incident
• Pricing
• Helicone on GitHub
• Helm charts repository
• Docker Hub image

Part two of three. A cumulative review of Beginner and Mid-level material for someone who runs Helicone at work, organised as scenarios and trade-offs rather than definitions. Expect an interviewer to ask what you would check first, then why, then what you would change. Senior reviews all three levels and adds platform ownership.

One framing point helps in every answer. Helicone has been in maintenance mode since it joined Mintlify on 3 March 2026: new sign-ups are disabled, security fixes and new models still ship, and the services remain live. A strong candidate mentions that status unprompted when asked about adoption, because it changes the risk discussion from "is this the best tool" to "how do we use it safely and leave it cleanly if we need to".

Where this picks up

The model in one screen

Helicone is an open-source LLM observability platform plus gateway. It records every request and response with tokens, cost, latency and metadata, and adds gateway features such as caching, rate limits, retries and provider failover, switched on per request with HTTP headers. In proxy integration it is on the critical path and every feature works. In async integration your app calls the provider directly and ships logs afterwards, so it is off the critical path but caching, rate limits and retries are unavailable.

Have these facts ready, because they come up in the first five minutes. The AI Gateway lives at https://ai-gateway.helicone.ai, is OpenAI-compatible, and takes the Helicone key as the bearer token. Legacy proxies (oai.helicone.ai, anthropic.helicone.ai, gateway.helicone.ai) need the provider key in Authorization plus Helicone-Auth: Bearer .... Header values are always strings. Bring-your-own-key is the default setup because pass-through billing needs a feature flag that most accounts do not have.

Scenarios

"Costs doubled last week and nobody changed anything. What do you check first?"

Start by bounding the question with the tools you have, not with guesses. Open the cost view and group by model, then by Feature and Tenant properties, and compare this week against last. There are four usual causes. A new or changed feature generating more calls. One tenant in a loop or running a bulk job. A model change, for example a routing string that now resolves to a more expensive model or a failover that moved traffic to a pricier provider. Or a cache that stopped hitting because someone added a timestamp to the prompt, which shows up as call volume unchanged and hit rate collapsing.

Then check the things that affect measurement, because "costs doubled" is sometimes "we started measuring correctly": streaming calls that previously logged zero cost because usage was not requested, or a custom model name newly resolved. Cross-check one day of logged cost against the provider's invoice to make sure Helicone and reality agree before you act.

A strong answer names the dimensions (feature, tenant, model, provider, version) and says the investigation is only possible because those properties were set in advance. A weak answer says "look at the dashboard" without saying what to group by, or proposes caching everything as the first fix.

"Cache hit rate is 2% on a route we expected to be highly repetitive."

The cache key is a hash of the seed, URL, full body, relevant headers including Authorization, and bucket index. Any volatile field in the body yields a new key. So I would capture two "identical" request bodies and diff them. Typical culprits are a timestamp or request ID in the system prompt, a message list with an extra trace message, a temperature that differs between callers, a user field in the body, and JSON key order from different serialisers. Fixes: remove the volatile field at source, or list it in Helicone-Cache-Ignore-Keys.

Then check the boring things: is the header value the string "true", is the seed different per call, is the route going through the gateway at all rather than async logging, and is the TTL so short the entries expire before the repeat arrives. The default is seven days and the maximum 365.

A strong answer also asks whether the route should be cached. Personalised answers, time-sensitive answers and variety-seeking chat do not benefit, and a bucket size above 1 returns random stored responses, which is deliberate non-determinism.

"A provider had an outage. What happened to our traffic, and what should have happened?"

It depends on how the call was made. With the model as a bare name, the gateway routes across providers offering it, cheapest first, and fails over on 429, 401, 400 for context length, 408 and 5xx, so most traffic should have moved without an error. With a pinned model such as gpt-4o-mini/openai, there is no failover, so the outage is our outage, and retries on 5xx are the only mitigation. With async logging, Helicone does nothing for us at all.

The follow-ups show depth. Which routes are pinned, and why (residency is legitimate, habit is not)? Did we notice the provider mix shift on the dashboard? Does a 401 failover mean a revoked key could be hiding for weeks, because traffic silently goes elsewhere? And what do the user-facing timeouts look like when retries multiply latency?

"Helicone itself is down. What breaks?"

Separate the two halves. The proxy (edge worker) is on the critical path: if it is unavailable, gateway calls fail, unless the client has a fallback path. The logging pipeline and dashboard are off the critical path: in the May 2026 AWS account incident, log ingestion and the dashboard were down for about four days while the proxy kept serving, and logs were queued without loss.

So the design question is whether the observability tool may take the application down. Options: accept the risk for the benefits of caching and failover; add a client-side fallback that calls the provider directly if the gateway is unreachable; or use async integration for paths where the dependency is unacceptable. A strong answer says which features that choice costs (caching, rate limits, retries) and mentions that in maintenance mode a longer-term exit path is part of the availability story.

"A customer asks where their prompts are stored. What do you say?"

Split by feature. Logged bodies go to Helicone's storage (object storage for bodies, ClickHouse for metadata), in the US or EU region of the organisation. Cached responses sit in Cloudflare Workers KV at the edge for the TTL, which is not in your infrastructure. Helicone-Omit-Request and Helicone-Omit-Response stop bodies being stored, but they still transit Helicone's backend and are held in memory. With the async logger, disable_content_tracing stops bodies being sent at all. For strict residency, pin to a regional BYOK deployment, avoid caching sensitive routes, and consider self-hosting for a single team, accepting its limits (see below).

"Per-user rate limiting is not working. One user made 500 calls against a 100-call policy."

Check in order. Does the request carry Helicone-User-Id? Segmentation with s=user silently does nothing without it, and s=<property> needs the matching Helicone-Property-<name>. Is the policy header well-formed, for example 100;w=3600;s=user, with a window of at least 60 seconds? Is the traffic going through the gateway, since limits do not exist under async logging? Is the header a string? And remember the limits are an edge guardrail backed by Workers KV, not an exact ledger, so brief overshoot is plausible under concurrency.

"Write me an 'agent run' trace you could debug in five minutes."

One Helicone-Session-Id per run, a Helicone-Session-Name for the run type, and paths by function: /plan, /plan/search, /plan/search/summarise, /answer. Not by time or step number, because plans change. Tool and retrieval spans get logged into the same session with the logger SDK so the slow step is visible even when it is not a model call. Plus the request ID in your application logs so one identifier links both worlds.

"Someone wants to self-host Helicone for a team pilot. Advice?"

Yes for an internal pilot, with eyes open. Use the all-in-one image with the three volumes mounted, one origin for all URL variables, NEXT_PUBLIC_IS_ON_PREM=true, a real BETTER_AUTH_SECRET and changed MinIO credentials, and Postgres and ClickHouse unreachable from outside. The limits: the image runs the dashboard and Jawn only and cannot proxy LLM traffic since the Jawn proxy was removed in August 2026, so you log through async or the manual logger. The newest image is from August 2025 and predates the 2026 security fixes, so it stays off the public internet. There is no published upgrade path and no maintained release tag. That is a lab or single-team tool, not a platform.

Trade-off questions

Gateway or async logging?

Gateway gives you the full feature set and a one-line model swap, at the price of a dependency on the critical path. Async gives independence and works with providers or SDKs the gateway does not cover, at the price of losing caching, rate limits and retries. Mixed is fine, decided per call site. A strong answer lists the features lost, not just "async is faster".

BYOK or pass-through billing?

BYOK: you keep your provider contracts, rate limits and invoices, and you control residency, but you rotate and store keys in Helicone's Provider Settings. Pass-through: one wallet and "0% markup", but since 25 August 2026 it needs the ptb_enabled flag, which most accounts lack, and a failed flag lookup denies it. In practice BYOK is the dependable default.

Automatic routing or pinning?

Automatic (gpt-4o-mini) is cheapest-first with load-balancing among equals and instant failover. Pinning (gpt-4o-mini/openai) buys determinism, residency and contractual clarity and forfeits failover. Reasoning: pin where something outside engineering requires it, automate elsewhere, and add retries to pinned routes.

Retries or failover?

Failover tries a different provider and is usually better. Retries repeat against the same provider and suit pinned calls and legacy proxies. Retries fire on 429 and 5xx only, never on other 4xx, and multiply worst-case latency, so size them from the latency budget.

Edge rate limits or application rate limits?

Edge limits stop spend before the provider call and need no code, but they depend on headers you set and are not exact. Application limits are authoritative and can express anything, but cost engineering effort and run after you have already decided to call. Use both: edge as the runaway-cost backstop, application for entitlements.

Helicone prompts or prompts in git?

Helicone gives environments, versions and non-engineers editing; git gives review, history and independence. Keep the authoritative copy in git and mirror into Helicone, particularly now that the product is in maintenance mode and export tooling covers logs much better than template history.

Keep Helicone or migrate?

Not a yes or no. Maintenance mode means security fixes and new models still ship and the services remain live with no published end date, so there is no emergency. But you should make leaving cheap: portable metadata schema, prompts in git, the client behind one function, scheduled exports. If you need new capabilities, or the security posture of the hosted product matters to you, migrating to an actively developed tool becomes a planned project rather than a scramble.

What a strong answer covers that a weak one misses

Common interview questions

1. What problem does Helicone solve?Visibility and control over LLM usage: every request logged with tokens, cost, latency and metadata, plus gateway features (cache, rate limits, retries, failover) switched on by headers. It answers who spent what, on which feature, and what failed.
2. Proxy versus async integration?Proxy sends traffic through Helicone and supports every feature, but it is on the critical path. Async calls the provider directly and ships logs, so it is independent but has no cache, limits or retries.
3. How do you authenticate to the AI Gateway versus a legacy proxy?Gateway: Helicone key as the bearer token only. Legacy: provider key in Authorization and Helicone-Auth: Bearer sk-helicone-.... Mixing them gives 401s that look like provider errors.
4. Explain the model string grammar.A bare model means every provider, cheapest first with failover. model/provider pins with no failover. model/azure/CUID targets a BYOK deployment. A comma list is an ordered chain, and a leading !provider excludes one.
5. When does failover trigger?On 429, 401, 400 for context length, 408 and 5xx. Models outside the registry route only through BYOK deployments. The docs are inconsistent about whether BYOK or pass-through is tried first, so test it.
6. Why do my retry headers do nothing?Most likely the values are numbers. All header values must be strings, so "3" not 3. Also retries only fire on 429 and 5xx.
7. How is the cache key built?Hash of the seed, full URL, full body, relevant headers including Authorization, and bucket index. One changed field in the body is a different key, which is why ignore keys exist.
8. What is a cache bucket?Up to N stored responses for one key, maximum 20. Later hits return one at random, which trades determinism for variety.
9. Where is cached data stored, and why does it matter?Cloudflare Workers KV at the edge, in Helicone's account, not your infrastructure. It matters for data-residency and retention answers.
10. Write a policy for five dollars per user per hour.Helicone-RateLimit-Policy: "500;w=3600;u=cents;s=user" with Helicone-User-Id set. The window must be at least 60 seconds, and a breach returns 429.
11. Why does a session need a path?The path builds the parent and child tree, so a trace shows structure. Design it by function and start with /, not by step number.
12. How do you attach a user rating to a request?POST /v1/request/{id}/feedback with {"rating": true}. Get the ID from the helicone-id header or pre-set Helicone-Request-Id.
13. Why was my score rejected?Scores accept integers or booleans only. A float such as 0.92 is refused, so send 92 and document the scale.
14. Streamed requests show wrong cost. Why?The usage chunk was not requested. Set stream_options={"include_usage": True} or the header helicone-stream-usage: "true".
15. The query API returns an empty array. Why?Property filters must be wrapped in request_response_rmt. Without it you get [] with no error.
16. What is HQL and what limits it?Read-only ClickHouse SQL over request_response_rmt: 300,000 rows, 30 seconds, 100 queries a minute, selected workspaces on Pro and above. Costs are stored times one billion. Since September 2026 it rejects quoted identifiers, SETTINGS and table functions.
17. How do you secure a webhook receiver?Verify helicone-signature, an HMAC-SHA256 over the JSON body, with a timing-safe compare. Acknowledge quickly because delivery times out at two minutes, and remember bodies truncate at 10 KB and the signed URL expires in 30 minutes.
18. What do the security and moderation headers cover?Both are OpenAI-only. Security uses Prompt Guard and optionally Llama Guard and returns PROMPT_THREAT_DETECTED. Moderation uses OpenAI's endpoint with your key and returns 400 PROMPT_FLAGGED_FOR_MODERATION.
19. What changed in August 2026 that affects old tutorials?The Jawn proxy and its /v1/gateway/* routes were removed, Experiments was removed, self-serve upgrades ended, and pass-through billing got a feature-flag gate.
20. Can a new learner sign up for Helicone Cloud?No, sign-ups were disabled with the March 2026 acquisition. Existing accounts work, and new learners can self-host the all-in-one image with its limits.
21. Why do self-hosted dashboards show "connection refused"?The browser is told localhost:8585. Set NEXT_PUBLIC_HELICONE_JAWN_SERVICE to a public URL and check __ENV.js.
22. How would you get your data out?npx @helicone/export with date, property and region filters, output JSON, JSONL or CSV, with checkpoint and resume. Schedule it, because retention caps what exists.

Questions to ask the interviewer

Good candidates ask as well as answer. Which integration style does the team use today, and is any traffic still on the legacy proxies? Are provider keys held in Helicone or elsewhere, and who can rotate them? How is cost reconciled against the provider invoice? What would it take for the team to leave Helicone, and has anyone rehearsed it? Those questions signal that you think about ownership and reversibility, which is the theme of the senior level.

Final self-test

• Say what each integration style gives up, and name which features need the proxy.
• Give the authentication rule for the AI Gateway and for a legacy proxy.
• Write three model strings: pinned, excluded and a manual chain.
• List the status codes that trigger failover and the ones that trigger retries.
• Explain the cache key and name two fixes for a low hit rate.
• Write a cost-based per-user rate-limit policy and say what must accompany it.
• Explain why "3" works and 3 does not.
• State what Helicone-Omit-Response does and does not do.
• Describe what broke and what stayed up in the May 2026 incident.
• Give three things that changed in 2026 and why they invalidate old posts.
• Say why a self-hosted install cannot proxy LLM traffic today.
• Describe how you would make leaving Helicone cheap.

Part two of three. At this level your calls are logged and the dashboard fills up, so the problems change character. They are no longer "why is nothing showing up" but "why is my cache useless", "why did the limit not fire", "why is this cost wrong" and "what do we do if Helicone goes away". Start with the error table, then the practices and practice cards beneath it, then the recipes.

A reminder that colours all of it: Helicone has been in maintenance mode since 3 March 2026. New sign-ups are disabled, old tutorials often describe removed features, and the docs still contain stale statements. When a tip here conflicts with a blog post, trust the tip, then check the current docs.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

Practice cards

1. Find your legacy integrationsSearch the repository for oai.helicone.ai, anthropic.helicone.ai, gateway.helicone.ai and Helicone-Auth. List each hit with the provider and owner. That is your migration backlog to the AI Gateway.
2. Read the headersSend one gateway request with curl and -i. Identify helicone-id, then repeat with caching on and find Helicone-Cache.
3. Break the string rule on purposeSend Helicone-Retry-Num as an integer from a typed client and confirm the retries do not happen. Then fix it and confirm they do.
4. Defeat your own cacheAdd a changing field to the body and watch the hit rate drop to zero, then rescue it with an ignore key.
5. Trip a per-user limitUse a tiny policy such as "2;w=60;s=user" with two user IDs and confirm each gets its own two calls.
6. Watch a failoverPut a deliberately invalid key for one provider in a test organisation and call a bare model name. Note which provider served the call.
7. Build a two-level sessionSend a parent call at /plan and a child at /plan/search with the same session ID, then open the tree.
8. Score a requestPre-set Helicone-Request-Id, post a boolean feedback rating and an integer score, and check both appear.
9. Verify a stream's costStream a call with and without include_usage and compare logged cost against the provider's usage page.
10. Verify a webhookReceive a delivery, recompute the HMAC, then change one byte of the secret and confirm rejection.
11. Rehearse the exitExport a day of metadata with @helicone/export, load it into SQLite, and recompute cost per feature.

Building a call policy module

Half the tips above disappear if header construction lives in one reviewed file. The module below turns scattered headers into named policies and keeps the string rule in one place.

from dataclasses import dataclass

def _as_header_value(v) -> str:
    """Helicone header values must be strings, including booleans and numbers."""
    if isinstance(v, bool):
        return "true" if v else "false"
    return str(v)

@dataclass(frozen=True)
class Context:
    user_id: str
    tenant: str
    feature: str
    route: str
    environment: str = "production"
    version: str = "unknown"

POLICIES = {
    "cached-summary": {
        "Helicone-Cache-Enabled": True,
        "Cache-Control": "max-age=86400",
        "Helicone-Retry-Enabled": True,
        "Helicone-Retry-Num": 2,
    },
    "interactive-chat": {
        "Helicone-RateLimit-Policy": "300;w=3600;u=cents;s=user",
    },
}

def build(ctx: Context, policy: str | None = None) -> dict[str, str]:
    headers = {
        "Helicone-User-Id": ctx.user_id,
        "Helicone-Property-Environment": ctx.environment,
        "Helicone-Property-Feature": ctx.feature,
        "Helicone-Property-Tenant": ctx.tenant,
        "Helicone-Property-Route": ctx.route,
        "Helicone-Property-Version": ctx.version,
    }
    if policy:
        headers.update(POLICIES[policy])
    return {k: _as_header_value(v) for k, v in headers.items()}

Because build converts everything, a caller who writes 2 instead of "2" cannot cause the silent failure. Pair it with a test that asserts on the output for each policy. The habit generalises: whenever a failure is silent, move the rule into code so the failure becomes impossible.

Operational recipes

See what the gateway did to one request

curl -si https://ai-gateway.helicone.ai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Helicone-Cache-Enabled: true" \
  -H "Helicone-Property-Environment: staging" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^HTTP\|^helicone'

Read the status line first, then helicone-id, then Helicone-Cache. If there is no helicone-id at all, the request probably never reached the gateway, so check the base URL.

Count requests by feature without the dashboard

curl -s https://api.helicone.ai/v1/request/query-clickhouse \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "request_response_rmt": {
        "properties": { "Feature": { "equals": "summarise" } }
      }
    },
    "limit": 50,
    "offset": 0
  }' | python3 -c 'import sys, json; d = json.load(sys.stdin); print(len(d.get("data", d)))'

If it prints zero, rerun with "filter": {} to prove data exists, then reintroduce the wrapper. For EU accounts replace the host with eu.api.helicone.ai.

Cost per tenant with HQL

SELECT
  properties['Tenant']      AS tenant,
  count()                   AS calls,
  sum(cost) / 1000000000.0  AS cost_usd
FROM request_response_rmt
WHERE request_created_at >= now() - INTERVAL 30 DAY
GROUP BY tenant
ORDER BY cost_usd DESC
LIMIT 25

Plain identifiers only, and request_response_rmt as the source. If your workspace lacks HQL, use the query API and sum in your own code.

Pin, exclude and chain providers

models = {
    "residency": "gpt-4o-mini/azure/ckx1example",          # pinned, no failover
    "no-openai": "!openai,gpt-4o-mini",                    # exclude a provider
    "ordered":   "gpt-4o-mini/azure,gpt-4o-mini/openai,gpt-4o-mini",
}

Keep the mapping in one place next to the policies, and document why each pin exists. A pin without a written reason is a future outage.

Score and rate a request from a background job

import os, requests

API = "https://api.helicone.ai"
H = {"Authorization": f"Bearer {os.environ['HELICONE_API_KEY']}"}

def score(request_id: str, relevance: float, grounded: bool):
    body = {"scores": {"relevance": round(relevance * 100), "grounded": grounded}}
    requests.post(f"{API}/v1/request/{request_id}/score",
                  headers=H, json=body, timeout=10).raise_for_status()

Scale floats to integers before sending and keep the scale (0 to 100) documented. Scores take roughly ten minutes to show up in aggregates.

Verify a webhook signature

import crypto from "node:crypto";

export function verify(body: unknown, signature: string, secret: string): boolean {
  const expected = crypto.createHmac("sha256", secret)
    .update(JSON.stringify(body))
    .digest("hex");
  const a = Buffer.from(signature);
  const b = Buffer.from(expected);
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

Check the digest encoding against a real delivery from your account before relying on this in production.

Stand up a local self-hosted instance

docker run -d --name helicone \
  -p 3000:3000 -p 8585:8585 -p 9080:9080 \
  -v helicone-postgres:/var/lib/postgresql/data \
  -v helicone-clickhouse:/var/lib/clickhouse \
  -v helicone-minio:/data \
  helicone/helicone-all-in-one:v2025.08.21

curl http://localhost:8585/healthcheck

# Verify a user's email by hand, because the container has no email service
docker exec -u postgres helicone psql -d helicone_test -c \
  "UPDATE \"user\" SET \"emailVerified\" = true WHERE email = 'you@example.com';"

Remember what this is for: a lab or a single-team dashboard. It cannot proxy LLM traffic, and the image predates the 2026 security fixes.

Export data on a schedule

HELICONE_API_KEY="sk-helicone-REPLACE_ME" npx @helicone/export \
  --start-date 2026-09-01 --end-date 2026-09-30 \
  --format jsonl --include-body

Drop --include-body when you only need metadata. A scheduled job in CI, running monthly with the output written to encrypted storage, turns "we could leave" into something tested. The GitHub Actions guide covers scheduled workflows.

Reading the dashboard critically

The dashboard is a view over data you labelled, so its usefulness is capped by your labels. Four habits keep it honest.

First, always filter out test traffic. If Environment is set consistently, a filter of production gives you the real picture; without it, a developer's load test is indistinguishable from a cost spike. Second, read latency as a distribution. A healthy mean can hide a bad p95 caused by retries, so look at the percentile and then at the status column for retried attempts, each of which is logged separately. Third, treat cost as an estimate until reconciled: it is computed from the provider's reported usage and Helicone's cost registry, so calls with missing usage or unknown model names are silently wrong. Fourth, remember retention. The plan determines how far back the data goes (7 days, one month, three months, or longer on enterprise), so a question about last quarter may be unanswerable if you never exported.

A good weekly review takes ten minutes: top tenants by cost, top features by cost, error rate by provider, cache hit rate on cached routes, and any alert that fired. If a number surprises you, ask which label would have explained it, and add that label.

Migrating a legacy call site to the AI Gateway

Most teams at this level inherit code that still points at a legacy proxy. Moving it is low-risk if you do it one call site at a time and change only what has to change. The work is smaller than it looks, because the gateway is OpenAI-compatible and most of the change is two lines.

Start by changing the base URL to https://ai-gateway.helicone.ai and the API key to the Helicone key, and delete the Helicone-Auth header, because on the gateway the Helicone key is the bearer token and a leftover header is harmless but confusing. Then make sure the provider key exists in Helicone's Provider Settings, since the gateway uses your stored key to call the provider, and your application no longer holds it. Forgetting this step is the usual cause of a 401 immediately after migration. Next, check the model string. A legacy call to gpt-4o-mini keeps working as a bare name, but it now means "cheapest provider first with failover", which may not be what a contract requires. If it was effectively pinned before, pin it now with the suffix.

Keep the existing metadata headers as they are, because Helicone-User-Id, Helicone-Property-* and the session headers behave the same on both paths. Anything that relied on the legacy Helicone-Fallbacks header should move to gateway routing, since fallbacks to unapproved domains were capped at 10,000 requests a day and one request a second. Finally, run the old and new paths side by side for a day if you can, tagged with a Transport property, and compare error rate, latency and cost. Only then remove the old code. Anthropic and Azure call sites follow the same pattern, with the provider key stored in Helicone rather than in the app.

Reasoning about cache, limits and cost together

The three features interact, and treating them separately produces surprises. A cache hit costs nothing at the provider and should not consume a cost-based rate-limit budget in any way you rely on, so do not assume it does without testing: a per-user budget of five dollars an hour behaves differently on a route with a ninety percent hit rate than on one with none. Retries multiply attempts, and each attempt is logged separately, so a burst of 5xx errors can inflate request counts and, for any attempt that does return tokens, real spend. Failover changes the price of a call because the cheapest provider for a model is not always the one you used yesterday.

The habit that keeps this manageable is to decide per route, in one table, what you want from each feature. Write the table in your policy module as data. For each route list whether it caches and for how long, whether it retries and how many times, whether it has a per-user or per-tenant limit, and whether it is pinned. Reviewing a pull request that changes that table is far easier than reviewing headers sprinkled across call sites, and it makes the interaction visible: a reviewer can see that a route with a long cache TTL also has a retry policy, and ask whether the combination makes sense.

When something is wrong in production

A compact order of operations helps when you are under pressure. First, decide whether the problem is your users' requests failing or only the telemetry being wrong. A failing request is an incident; a missing log is an annoyance unless it hides a real failure. Second, read the status and headers of one failing call, because that answers about half of all cases in seconds: a 401 is credentials, a 429 is either credits or your own policy, a 400 is a guard or a context-length problem, and a 5xx is the provider or the gateway. Third, check the provider mix and the error rate by provider, which tells you whether failover is absorbing the problem or whether a pin is exposing you to it.

Fourth, check whether anything changed recently: a deploy (the Version property lets you compare before and after), a prompt version promoted to production, a key rotated in Provider Settings, or a change on Helicone's side. For the last one, remember that the product still receives security fixes and new model updates, so behaviour can shift without a release tag. Fifth, if the telemetry and the dashboard are unavailable but requests succeed, do not panic: that was the shape of the May 2026 incident, where the proxy kept working and the queued logs were not lost. Write down what you need to know, wait for ingestion to recover, and use your own application logs and the provider's usage pages in the meantime.

Finally, write the incident up with the dimensions that mattered, because those tell you which property you wished you had. Every post-incident review at this level should end with one sentence of the form "we would have found this faster if requests had carried X", and X goes into the metadata wrapper the same week.

Habits worth adopting now

• Treat Helicone headers as an API you own, with a module, tests and a changelog.
• Write down why each provider pin exists.
• Put the helicone-id in your application logs.
• Keep prompts in git as well as in Helicone.
• Add include_usage to every streaming path.
• Review who created the production API keys every quarter.
• Keep an eye on the provider mix as an early warning for dead keys.
• Schedule an export and check that you can load it.
• Read the changelog-by-incident for the month before you trust a tutorial.

Checklist

• Every call goes through one client module and one header builder.
• All header values are strings, asserted by a test.
• Properties follow a written schema: Environment, Feature, Tenant, Route, Version.
• Provider keys are configured as BYOK, and I know who rotates them.
• Every pinned model has a written reason and retries enabled.
• Caching is on only for routes where repeats are expected, with ignore keys set.
• Per-user or per-tenant limits include the matching segmentation header.
• Streaming calls request usage.
• Alerts have minimum request counts and group by tenant.
• Webhook receivers verify signatures and acknowledge quickly.
• Cost is reconciled against the provider invoice each month.
• A scheduled export exists and a restore has been rehearsed.
• I know which legacy proxy integrations remain and who owns them.

This is part three of three. The beginner level got a request logged; the mid level put Helicone into production with sessions, properties, caching, routing and prompts. This level is for the person who owns Helicone as a platform that other teams depend on: the one who is paged when the gateway misbehaves, who answers the security questionnaire, who explains the invoice, and who decides whether the company should still be on Helicone next year.

That last question is not rhetorical. Helicone joined Mintlify in March 2026 and has run in maintenance mode since. New cloud sign-ups are disabled, the self-hosted image has not had a tagged release since August 2025, and in 2026 the team shipped a run of security fixes, including cross-tenant fixes, that the published image does not contain. None of that makes Helicone unusable. It does mean a senior owner has to reason about it as a platform with a known status, known gaps and a probable exit, rather than as a product that will keep growing to meet you. This guide treats it that way from the first section to the last.

Where this picks up

Every topic from the earlier levels comes back here with an ownership dimension attached. The table shows what changes.

I start with the status of the platform, because every later decision depends on it.

The status of the platform you are inheriting

A senior engineer inheriting a tool asks three questions before anything technical. Who maintains it? What ships, and how often? What happens to my data if it stops? For most tools in this catalogue, the answers are boring. For Helicone in late 2026 they are not, and you should be able to state them precisely.

Who maintains it. On 3 March 2026 Helicone announced it was joining Mintlify. The founders moved to Mintlify, and the announcement says Helicone continues "in maintenance mode", with "security updates, new models, bug & performance fixes" still shipping and services remaining live "for the foreseeable future". No end-of-life date has been published. That is a reasonable promise, and the commit history backs it up: the main branch of the monorepo kept receiving commits through September 2026, including significant security work. It is also explicitly not a promise of new features.

What ships. Helicone Cloud is continuously deployed, so cloud users receive fixes without doing anything. The self-hosted story is different. The newest tagged release of Helicone/helicone is v2025.08.21-1, and the newest Docker image, helicone/helicone-all-in-one:v2025.08.21 (which is also what latest points at), was published on 21 August 2025. Everything fixed after that date, including every 2026 security change described in this guide, is only available to self-hosters who build from main. The standalone Rust AI Gateway (Helicone/ai-gateway) stopped at v0.2.0-beta.30 in July 2025, changed licence from Apache 2.0 to GPL v3.0 in November 2025, and its documentation pages now return "Page Not Found". Treat it as abandoned.

What happens to your data. New sign-ups were disabled the same day as the announcement, so a Helicone Cloud account is now something you have rather than something you can get. Self-serve upgrades to paid plans were removed in August 2026; the upgrade UI points to a contact page. The founders say they will help customers migrate, and there is an official export tool, @helicone/export, covered later. In practice the market already treats Helicone as a platform to plan an exit from: practitioner write-ups in 2026 are titled things like "Moving off Helicone".

What does this mean for an owner? Three concrete things.

1. Your risk register needs an entry. Write down which applications depend on Helicone, whether each is on the critical path (proxy or gateway) or off it (async logging), what data lives in Helicone that you cannot recreate, and how long an exit would take. A platform in maintenance mode is not an emergency, but a platform whose owner has not written this down is a future emergency.
2. Do not build new capabilities that only Helicone provides. If a team wants to adopt a Helicone-specific feature today, ask what they will replace it with. Features that map cleanly to other tools (request logging, cost by property, sessions as traces) are low-risk. Features that do not (Helicone's prompt templating syntax, its HQL dialect) accumulate exit cost.
3. Watch the repository, not the blog. Security-relevant changes in 2026 arrived as pull requests with behaviour changes (removed routes, rejected query syntax, changed key semantics). Subscribe to the repository's releases and pull requests, and read them.

Architecture and its failure modes

You cannot reason about failure without a precise picture of the components. The mid level introduced the pieces; here is how they depend on each other, and what each one's failure looks like from the outside.

The request path in Helicone's cloud works like this. Your client sends an LLM request to the edge: either the AI Gateway at https://ai-gateway.helicone.ai or one of the legacy provider proxies. The edge is Cloudflare Workers code. If the request carries feature headers, the Worker consults Workers KV for cache entries and rate-limit counters, then forwards the request to the provider and streams the response back. Only after returning the entire response does it put a log message on Kafka. Consumers in Jawn read from Kafka and write metrics to ClickHouse, configuration-linked records to Postgres, and full request and response bodies to object storage. The dashboard and the REST API read from those stores through Jawn.

Two design rules from Helicone's own availability documentation shape the failure behaviour. First, the proxy carries minimal business logic unless feature headers are present. Second, feature code is wrapped so that on failure it falls back gracefully to "just proxying". In other words, if the cache lookup or the rate-limit check throws, your request is still forwarded, just without that feature.

Here is what each component's failure looks like from an application team's point of view.

That last row deserves honesty: the docs do not say exactly how the edge behaves if it cannot validate a key, so treat it as unknown and test your own fallback rather than relying on an assumption.

The May 2026 incident as a worked example

Between 14 and 18 May 2026, AWS locked Helicone's account after flagging a Bedrock key as compromised. According to Helicone's write-up, the key was not in fact compromised. For about four days, log ingestion and the dashboard were down. The proxy and gateway stayed fully functional, logs were queued, and none were lost. All credentials were rotated afterwards.

Read that through the table above and it is exactly the designed failure: the read path and the logging path went dark, the critical path did not. For an owner, the lessons are specific.

• Four days without a dashboard is four days without cost alerts, error-rate alerts and per-user usage data. If your only cost guardrail was a Helicone alert, you had no guardrail that week. Rate limits enforced at the edge, in contrast, kept working.
• Queued logs arrived later, so any downstream consumer that assumes near-real-time delivery (a billing job, a webhook-fed moderation queue) saw a gap followed by a burst. Design consumers to tolerate both.
• A vendor's cloud account is itself a dependency. Nothing in your architecture could have prevented this; your only defence is to decide in advance which Helicone functions you can live without for a week.

The critical-path decision: gateway, proxy or async

At beginner and mid level, the integration method was a setup choice. At senior level it is an availability decision, and it should be made per workload with a written reason.

There are three shapes. The AI Gateway (https://ai-gateway.helicone.ai, authenticated with your Helicone API key as the bearer token) is the recommended path: OpenAI-compatible, routing and failing over across providers, and logging. The legacy proxies (oai.helicone.ai, anthropic.helicone.ai, and gateway.helicone.ai with Helicone-Target-URL) pass requests through unchanged, with your provider key in Authorization and your Helicone key in Helicone-Auth. The docs call these "maintained but no longer actively developed". Async logging means your application calls the provider directly and ships logs to Helicone afterwards, through OpenLLMetry (helicone-async / @helicone/async), the manual logger (helicone-helpers / @helicone/helpers) or the raw logging API.

The argument for a proxy or gateway is that it is the only way to get the edge features: a response cache that saves money, rate limits that stop a runaway agent, provider failover that survives an OpenAI incident. The argument against is that you have added a network hop and a vendor to every request's success path. Helicone's own latency benchmark reports mean latency of 2.21 s both with and without the proxy, p90 of 3.29 s versus 3.27 s, and max of 3.76 s versus 3.56 s. It was measured in the text-ada-001 era with 500 interleaved requests, so treat it as a vendor claim about added latency being small, not as a number for your workload.

A useful rule for a platform owner: a workload should be on the critical path only if it uses at least one edge feature. A batch summarisation job that uses no caching, no rate limits and no failover gets nothing from the proxy except risk; move it to async logging. A customer-facing chat service that relies on per-user cost limits needs the edge, and should be designed with a client-side fallback.

Designing the fallback

If a service must use the gateway, it should still be able to call the provider directly when the gateway is unreachable. This is a circuit breaker in your client code: try the gateway, and on connection errors or gateway-originated 5xx, call the provider directly with your own key for a bounded period. The cost is that during the fallback you lose logging and edge features, so the fallback should itself log that it is active.

import os
import time
from openai import OpenAI, APIConnectionError, InternalServerError

GATEWAY = OpenAI(
    base_url="https://ai-gateway.helicone.ai",
    api_key=os.environ["HELICONE_API_KEY"],
)
DIRECT = OpenAI(api_key=os.environ["OPENAI_API_KEY"])  # break-glass path

_open_until = 0.0  # circuit breaker: while now < _open_until, skip the gateway


def chat(messages, *, model="gpt-4o-mini", headers=None, cooldown=60):
    global _open_until
    if time.monotonic() >= _open_until:
        try:
            return GATEWAY.chat.completions.create(
                model=model, messages=messages, extra_headers=headers or {}
            )
        except (APIConnectionError, InternalServerError):
            _open_until = time.monotonic() + cooldown
            print("helicone gateway unavailable; direct mode for", cooldown, "s")
    # Direct mode: no logging, no cache, no Helicone rate limits.
    # Pin the provider model explicitly; gateway suffixes like "/openai" are not valid here.
    return DIRECT.chat.completions.create(model=model.split("/")[0], messages=messages)

Three details in that file are the lesson. The break-glass path needs its own provider key, which means the key exists outside Helicone as well as inside it; that is a custody decision you make on purpose. Gateway model strings such as gpt-4o-mini/openai or !openai,gpt-4o-mini are Helicone grammar, not provider grammar, so the fallback must translate them. And an InternalServerError might be the provider's own 500 passed through, in which case switching to direct mode does not help; in production you would distinguish them by checking for Helicone response headers such as helicone-id.

Scaling: which component breaks first

"Does Helicone scale?" is the wrong question. The useful question is which limit you will hit first, and the answer is different for the cloud and for a self-hosted install.

On Helicone Cloud: plan limits, then query limits

On the cloud, your first ceilings are contractual rather than technical. The published plans cap log ingestion rates: Hobby 10 logs per minute, Pro 1,000, Team 15,000, Enterprise 30,000. API access is capped at 10, 60 and 1,000 calls per minute across the tiers. Retention is 7 days on Hobby, one month on Pro, three months on Team, and forever or configurable on Enterprise. Because self-serve upgrades have been removed, raising any of these now means a conversation with the vendor rather than a click, so know where you stand before you need headroom.

Retention is the limit people discover last and regret most. Three months of data on the Team plan means that "how did cost per conversation change year on year?" is unanswerable from Helicone alone. If you need long-term history, export it on a schedule to storage you own. The export CLI supports date ranges and resume, which makes a monthly job straightforward.

Next come the query-side limits, which bite platform teams building reports.

HQL is also access-controlled: it is available on selected workspaces (you request access) and on the Pro plan and above.

Self-hosted: storage fills before compute saturates

On a self-hosted install, the recommended single node is 4 vCPU, 16 GB RAM and 100 GB disk (an AWS m5.xlarge-class machine). On that shape, the component that breaks first is almost always disk, and specifically the object store and ClickHouse, not CPU. Every logged request writes its full request and response body to MinIO, and LLM bodies are large: a retrieval-augmented request with ten context chunks can be tens of kilobytes before the answer. Multiply by request volume and there is no retention job in the all-in-one image doing the deleting for you.

The order in which things usually go wrong on a busy single node:

1. MinIO disk fills. Body writes fail; metadata may still land in ClickHouse, so the dashboard shows requests with missing bodies.
2. ClickHouse disk and merges. request_response_rmt grows steadily; background merges compete with dashboard queries, and slow dashboards appear before failures do.
3. Postgres is rarely the bottleneck: it holds organisations, keys and prompts, which grow with configuration rather than traffic.
4. Jawn CPU becomes relevant only at high ingestion rates, and scales horizontally once you move to Kubernetes.

For anything beyond a single node, the Helm charts in Helicone/helicone-helm-v3 split the system into helicone-core (web, Jawn, worker), helicone-infrastructure (PostgreSQL, Redis, ClickHouse, autoscaling, Beyla), helicone-monitoring (Grafana, Prometheus) and helicone-argocd. Terraform modules in the same repository provision EKS, S3 and Aurora; with Aurora you set postgresql.enabled=false, and with MinIO disabled you set s3BucketName, s3Endpoint, s3AccessKey and s3SecretKey and add an S3 CORS rule that allows GET from the web origin, because browsers fetch bodies through presigned URLs. The chart repository has no tagged releases and its last push was in September 2025, so pin to a commit and read what you deploy.

git clone https://github.com/Helicone/helicone-helm-v3.git
cd helicone-helm-v3
git rev-parse HEAD                      # record the commit you are deploying
cd helicone && helm dependency build
cp values.example.yaml values.yaml
cp secrets.example.yaml secrets.yaml    # never commit this file
helm upgrade --install helicone-core ./helicone-core -f values.yaml -f secrets.yaml
kubectl get pods

Security and the trust model

A trust model answers one question for every party: what can they do if they are malicious or compromised? For Helicone, there are five parties worth naming.

Your application holds a Helicone API key (sk-helicone-...). Since June 2024, every Helicone API key has write access; earlier documentation distinguished read and write keys, and some older internal runbooks still assume that distinction exists. A leaked key can therefore send traffic billed to your provider keys through the gateway, read your logged requests, including bodies, through the REST API, and write feedback, scores and properties. Treat it as a production credential, not as a telemetry token.

Helicone holds your provider keys if you use the AI Gateway with BYOK ("bring your own key", configured in Provider Settings). The gateway calls OpenAI, Anthropic and others on your behalf using those keys. Helicone says provider keys are stored encrypted, but the documentation page that described the storage in detail now returns 404, so do not quote specifics in a security review. On the legacy proxies you keep custody: the provider key travels in each request's Authorization header and is not stored.

Cloudflare runs the edge. Cached responses live in Cloudflare Workers KV, which is Helicone's infrastructure on Cloudflare's network rather than yours. For many teams that is fine; for a team with data-residency obligations it is a finding.

Other Helicone tenants share the cloud. In 2026 Helicone fixed several cross-tenant problems: a module-level OpenAI client in the dashboard's API routes leaked one organisation's provider key to other organisations; an HQL validation gap allowed a cross-tenant bypass; and a path allowed platform-admin takeover. All are fixed in the cloud. The lesson for an owner is not that Helicone is uniquely careless, but that a multi-tenant observability platform holding provider keys and prompt bodies is a high-value target, and its tenancy boundary is software you do not control.

Other members of your own organisation can create keys. Since the September 2026 security change, an API key created by a non-owner member carries that member's user_id, and if the member leaves the organisation the key fails membership checks. That is a sensible security fix, and it is also an outage waiting to happen: the day a departing engineer's account is removed, every service using a key they created stops authenticating.

Self-hosters carry the 2026 exposure

Everything above describes the cloud after the fixes. A self-hosted install on v2025.08.21 has none of them. That includes the March 2026 removal of unauthenticated endpoints that ran SQL, the February and March 2026 batch of fixes (parameterised ClickHouse queries, DOMPurify sanitising, and stripping dangerous HTML from LLM output to stop stored XSS in the dashboard), the August 2026 tenancy fixes, and the September 2026 HQL lexer and admin-takeover fixes. If your self-hosted install is reachable by anyone you do not fully trust, you are running software with publicly described vulnerabilities.

Your options are to build from main (and own that build, as the supply-chain section describes), to restrict network access so that only trusted operators reach the dashboard and Jawn, or both. Network restriction is the minimum: Postgres (5432) and ClickHouse (8123) should never be reachable from outside the host, and ports 3000, 8585 and 9080 should sit behind TLS and, ideally, behind your SSO-aware reverse proxy or VPN.

The self-hosted defaults need changing on day one: BETTER_AUTH_SECRET defaults to change-me-in-production, and the MinIO credentials S3_ACCESS_KEY and S3_SECRET_KEY default to minioadmin / minioadmin. Docs written about the old Jawn proxy also carried a warning that still applies to any self-hosted gateway: the proxy port "does not require authentication for proxying requests. Anyone with access can proxy LLM requests through your endpoint."

What screening does and does not cover

Helicone offers two request-screening features at the edge. Helicone-LLM-Security-Enabled: "true" runs Meta's Prompt Guard (86M parameters) to detect jailbreaks and direct or indirect prompt injection in eight languages; adding Helicone-LLM-Security-Advanced: "true" also runs Llama Guard (3.8B) across 14 categories. A blocked request returns an error with code PROMPT_THREAT_DETECTED. Separately, Helicone-Moderations-Enabled: "true" calls OpenAI's moderation endpoint on the latest user message, using your OpenAI key configured in Helicone, and a flagged prompt returns HTTP 400 with code PROMPT_FLAGGED_FOR_MODERATION.

Both features work only for OpenAI models. If your routing sends a request to Anthropic or Gemini, the screening you think is on is not happening. Both are also opt-in per request, which means any caller who omits the header is unscreened. Screening at the gateway is a useful layer; it is not a guardrail policy. If your organisation needs guaranteed screening, enforce it in a component you control, such as the shared client library described later, or a dedicated tool such as the ones in the Guardrails guide.

Omitting bodies is not the same as not sending them

Helicone-Omit-Request and Helicone-Omit-Response stop Helicone from storing bodies. The docs are explicit that bodies are still sent to Helicone's backend and held "only in memory". For a privacy review, that means the data still transits a third party. If a workload may not send personal data to Helicone at all, the proxy and gateway are the wrong integration, and async logging with content tracing disabled (logger.disable_content_tracing() in helicone-async) is the right one, because then only metrics leave your process.

Data residency for Gulf and Egyptian employers

Helicone Cloud offers two regions: US (us.helicone.ai, API api.helicone.ai) and EU (eu.helicone.ai, API eu.api.helicone.ai). The documentation lists no Middle East region. Teams in Saudi Arabia, the UAE, Qatar or Egypt whose legal teams require that prompts and completions stay in-country, or that personal data leave only under specific conditions, cannot meet that with Helicone Cloud, and the edge cache in Cloudflare KV complicates even an EU-only argument. Check with your own counsel what your obligations actually are; this guide is not legal advice.

The workable pattern is to self-host in a cloud region inside the jurisdiction (the major clouds all operate Gulf regions) and use async logging, since there is no documented working self-hosted gateway. You lose the edge features, but prompts and completions never leave your infrastructure, and the provider call is governed by whatever arrangement you have with the provider itself.

Secrets and key custody

Four families of secret exist around a Helicone deployment, and each has a different owner, a different rotation story, and a different blast radius when it leaks.

The Helicone API key is the one your application teams handle daily, so it is the one most likely to leak. The rotation procedure is the standard two-key overlap: create the new key, roll it out everywhere, confirm from the dashboard that traffic is arriving under the new key, and only then delete the old one. Deleting first turns a rotation into an outage.

BYOK moves provider-key custody into Helicone, which simplifies application code (applications only hold the Helicone key) and concentrates risk (one Helicone compromise exposes every provider key). The 2026 provider-key leak between organisations is the concrete example of that concentration. For the most sensitive provider accounts, a defensible pattern is BYOK for general traffic, with a separate provider key, a separate provider project and a hard spending cap at the provider, so that the worst case is bounded by something Helicone cannot change.

Self-hosters on the August 2026 code or later also have a database step tied to key custody. The tenancy fix added a composite foreign key (created NOT VALID) linking proxy keys to provider keys. The migration header contains an audit query; you run it, resolve any rows it reports, and then validate the constraint so that Postgres enforces it for existing rows as well as new ones.

-- After running the audit query from the migration header and fixing its findings:
ALTER TABLE <proxy_key_table> VALIDATE CONSTRAINT <constraint_name_from_migration>;

Take the exact table and constraint names from the migration file in the commit you are deploying; they are deliberately left as placeholders here because copying names from a guide into a production database is how constraints end up validated on the wrong table.

Verifying webhooks properly

Webhooks carry request data out of Helicone to an endpoint you run, so an attacker who can post to that endpoint can inject fake events into whatever consumes them: an evaluation queue, a billing job, a moderation review tool. Helicone signs each delivery with HMAC-SHA256 and sends a signature in the helicone-signature header (the example below assumes hex encoding; confirm the encoding against a real delivery). The docs' reference implementation computes the HMAC over JSON.stringify(payload).

import crypto from "crypto";
import express from "express";

const secret = process.env.HELICONE_WEBHOOK_SECRET!;
const app = express();
app.use(express.json({ limit: "1mb" }));

function verify(payload: unknown, signature: string): boolean {
  const expected = crypto
    .createHmac("sha256", secret)
    .update(JSON.stringify(payload))
    .digest("hex");
  const a = Buffer.from(expected, "hex");
  const b = Buffer.from(signature ?? "", "hex");
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

app.post("/helicone/webhook", (req, res) => {
  if (!verify(req.body, req.header("helicone-signature") ?? "")) {
    return res.status(401).end();
  }
  res.status(202).end();                // acknowledge well inside the 2-minute timeout
  queueForProcessing(req.body);         // fetch request_response_url within 30 minutes
});

function queueForProcessing(event: any) { /* hand off to your queue */ }

app.listen(8080);

Three details matter. The length check before timingSafeEqual is needed because that function throws on buffers of different lengths, and an attacker controls the header. Because the signature covers JSON.stringify of the parsed body, a receiver in another language must reproduce the same serialisation, key order and spacing included, or verification fails on legitimate payloads; Node is the path of least resistance. And the handler acknowledges before processing, because delivery times out after two minutes and the full-body URL expires after thirty.

The self-hosted supply chain

Most tools in this catalogue let a self-hoster pin a release, read its changelog and upgrade. Helicone's self-hosted story in 2026 is harder, because the release that exists is old and the fixes that matter are only on main. A senior owner has to choose deliberately between two unattractive options, and should write down which one they chose and why.

Option one: run the published image. You get a known artifact, helicone/helicone-all-in-one:v2025.08.21, and you accept the 2026 vulnerabilities. This is defensible only if the install is reachable exclusively by trusted operators over a private network, holds no data you would mind an insider reading, and is treated as disposable. Pin it by digest, not by latest, so that nothing changes underneath you.

docker pull helicone/helicone-all-in-one:v2025.08.21
docker image inspect helicone/helicone-all-in-one:v2025.08.21 --format '{{index .RepoDigests 0}}'
# Deploy by the digest this prints, never by :latest

Option two: build from main. You get the security fixes, and you become the release engineer for someone else's monorepo. That means choosing a commit, building the images yourself, running the Postgres migrations with flyway, validating the new constraint, and testing that the dashboard, ingestion and your integrations still work. There is no published upgrade guide. The manual self-hosting guide describes the development setup (Node 20 via nvm, yarn, Python helpers for ClickHouse and MinIO, Postgres 17.4 and flyway migrate -configFiles=supabase/flyway.conf), which tells you the moving parts but is not a production build process.

If you choose option two, run it like any internal fork:

1. Pin a commit, record its hash in your deployment repository, and move it forward on purpose, never automatically.
2. Build images in your own CI, push them to your registry, and deploy by digest. Scan them like any other image.
3. Read the diff of every pull request between your old and new commit that touches valhalla/jawn, web, or the migrations. The 2026 changes removed routes and changed API semantics; a blind upgrade breaks integrations.
4. Back up Postgres, ClickHouse and object storage before migrating, and practise the restore once so you know it works.
5. Canary: upgrade a staging install fed with mirrored async logs first.

The client-side packages are part of the supply chain too. The npm and PyPI helpers were last published in late 2025 (@helicone/helpers 1.8.3, helicone-helpers 1.2.1, @helicone/async 2.0.2, helicone-async 1.1.1, @helicone/mcp 0.1.6, @helicone/export 1.0.0). Pin exact versions in lockfiles. In particular, the docs' MCP configuration runs npx @helicone/mcp@latest, which fetches whatever is newest at every launch; on a developer laptop holding a production Helicone key, pin the version instead.

{
  "mcpServers": {
    "helicone": {
      "command": "npx",
      "args": ["@helicone/mcp@0.1.6"],
      "env": { "HELICONE_API_KEY": "${HELICONE_READONLY_PURPOSE_KEY}" }
    }
  }
}

Note the environment variable name in that file: it signals intent, not a capability. Every Helicone key has write access, so there is no truly read-only key to give an MCP client. Create a dedicated key for it so you can revoke it alone.

Multi-tenancy: your tenants inside Helicone

There are two meanings of multi-tenancy here, and mixing them up causes real incidents. The first is Helicone's tenancy: organisations on the cloud, which you do not control. The second is yours: the customers, business units or internal teams whose traffic flows through your Helicone organisation. This section is about the second.

Helicone gives you three tools for tenant separation inside one organisation: Helicone-User-Id for the end user, custom properties such as Helicone-Property-Tenant for the customer or team, and separate Helicone organisations for hard separation. Organisations are limited by plan (one on Hobby, five on Team, unlimited on Enterprise), so most platforms separate tenants with properties and reserve organisations for boundaries that must be hard, such as production versus development, or a regulated business unit.

With a Tenant property in place, you get per-tenant cost and usage in the dashboard, per-tenant alerts (alerts can group by property), per-tenant rate limits (s=tenant in the policy, with the matching property header), per-tenant webhook routing (webhook property filters, all of which must match), and per-tenant HQL queries.

headers = {
    "Helicone-User-Id": user.id,
    "Helicone-Property-Tenant": tenant.slug,
    "Helicone-RateLimit-Policy": "2000;w=3600;u=cents;s=tenant",  # $20 per hour per tenant
    "Helicone-Cache-Enabled": "true",
    "Helicone-Cache-Seed": tenant.slug,
}

The last header is the one people forget, and it is a data-exposure bug rather than a performance detail. Helicone's cache key is a hash of the seed, the full URL, the full body, relevant headers including Authorization, and the bucket index. On the AI Gateway, Authorization carries your Helicone API key, which is the same for every tenant of your application. So if two tenants send byte-identical requests, which is common for a templated prompt with a short user question, the second tenant can receive the first tenant's cached answer. If that answer contains anything tenant-specific (a name retrieved into context, an account detail the model repeated), you have leaked data between customers. Always seed the cache per tenant, and per user where answers are personal.

Tenant data separation has one more dimension: who in your company can read which tenant's logged bodies. Helicone's organisation membership has no row-level permission by property, so anyone in the organisation sees every tenant's traffic. If some tenants' data must be visible only to a restricted group, those tenants need their own organisation, or their bodies should be omitted.

Cost: attribution, control and the 1e9 unit

Cost is where Helicone earns its keep for most organisations, and where a platform owner is most often asked hard questions. The work divides into measuring cost correctly, attributing it, and controlling it.

Measuring correctly

Helicone computes cost from the provider's usage block and its open-source cost registry of 300+ models. Two situations produce wrong numbers. Streaming responses omit usage unless you ask for it, so set stream_options={"include_usage": True} or send helicone-stream-usage: "true"; without it, streaming cost can be wrong or zero. And models whose names Helicone cannot resolve (custom deployments, some Azure deployment names) need Helicone-Model-Override so the registry can price them. A platform owner should check both across all teams, because a single team streaming without usage makes the whole organisation's cost report quietly wrong.

Attributing with HQL

HQL (Helicone Query Language) is read-only ClickHouse SQL over the request_response_rmt table, executed through POST /v1/helicone-sql/execute. It is where showback reports come from. The one fact you must not forget: cost columns are stored multiplied by 1,000,000,000, so divide by 1e9 to get US dollars.

SELECT
  properties['Team']                AS team,
  request_model                     AS model,
  count()                           AS requests,
  sum(total_tokens)                 AS tokens,
  round(sum(cost) / 1000000000, 2)  AS usd
FROM request_response_rmt
WHERE request_created_at >= now() - INTERVAL 30 DAY
GROUP BY team, model
ORDER BY usd DESC
LIMIT 50

curl -s https://api.helicone.ai/v1/helicone-sql/execute \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{"sql": "SELECT request_model, round(sum(cost) / 1000000000, 2) AS usd FROM request_response_rmt WHERE request_created_at >= now() - INTERVAL 7 DAY GROUP BY request_model ORDER BY usd DESC LIMIT 20"}
JSON

Since the September 2026 security change, the HQL validator rejects backtick- or double-quoted identifiers, SET and SETTINGS, heredocs, # comments, multiple statements, database-qualified table names and ClickHouse table functions such as numbers(). FROM and JOIN targets must be request_response_rmt or a parenthesised subquery. Saved queries written before that date may now fail; if a team's report broke in September 2026, check for those constructs first. Single-quoted string literals, as in the map lookup above, are fine.

Controlling spend

Four levers exist, and they work at different speeds.

The combination that holds up is edge limits for fast protection, alerts for awareness, and provider caps as the last line. Alerts support thresholds over a time frame, a minimum request count, grouping by user, model, provider or property, and aggregations including percentiles, delivered by email or Slack. A cost alert grouped by Team with a sensible minimum request count catches the "someone left a loop running" incident without paging on single expensive requests.

Upgrades and migrations

Three kinds of migration land on a Helicone owner in 2026: moving code off deprecated Helicone surfaces, keeping a self-hosted install current, and moving off Helicone altogether. They need different plans.

Off deprecated surfaces. The legacy proxies still work but are no longer developed; the Jawn /v1/gateway/* routes and the OpenAI Realtime WebSocket through Jawn were removed in August 2026; Experiments were removed at the same time (tables and historical data kept); the Helicone-Prompt-Id header and hpf/hpstr template helpers are replaced by prompt_id and inputs in the request body. Moving a service from a legacy proxy to the AI Gateway changes three things at once: the base URL, the credential (only the Helicone key, as the bearer token), and the model string. Current docs use model: "gpt-4o-mini" with a suffix for pinning (gpt-4o-mini/openai); the old Rust gateway README used provider/model (openai/gpt-4o-mini) with a /ai base path. Mixing the two grammars is the most common migration bug, so grep for both before declaring a service migrated.

Keeping self-hosted current. The procedure from the supply-chain section applies: pin, back up Postgres, ClickHouse and object storage, test on staging, roll. Treat every upgrade as a migration, because there is no upgrade guide and no semantic versioning to tell you what broke.

Moving off Helicone. A maintenance-mode platform deserves a written exit plan even if you never execute it. The plan has three parts.

1. Export the history you need. @helicone/export pulls requests with checkpoint and resume, retries with backoff, and writes JSON, JSONL or CSV. Export is a paid-plan feature. Run it by month so a failure costs one month, not a year.

HELICONE_API_KEY="$HELICONE_API_KEY" npx @helicone/export@1.0.0 \
  --start-date 2026-08-01 --end-date 2026-08-31 --format jsonl --include-body
HELICONE_API_KEY="$HELICONE_API_KEY" npx @helicone/export@1.0.0 \
  --region eu --start-date 2026-08-01 --end-date 2026-08-31 --include-body

2. Replace the integration without a big bang. Because every integration is a base URL plus headers, the cleanest cut-over is in the shared client library: add the new destination behind a flag, dual-log for a period (the gateway for traffic, plus the new tool's async logger), compare the numbers, then flip. The practitioner write-ups name Langfuse, the LiteLLM proxy, OpenRouter and Portkey as common destinations; each replaces a different subset of Helicone, which the next section maps.
3. Replace the Helicone-only features. Prompts stored in Helicone need exporting and re-templating (the {{hc:name:type}} syntax is Helicone's own). Saved HQL queries need rewriting for the destination's query layer. Alerts and webhooks need re-creating. Inventory these first; they are what makes an exit take months rather than weeks.

Incident playbooks

Each playbook starts with the question that most quickly narrows the problem, because under pressure ordering matters more than knowledge.

Logging is down, traffic is fine. This is the May 2026 shape. Do not break glass: the proxy works and edge limits still protect you. Tell finance and the teams that dashboards and alerts are blind, switch to provider-side usage reports for cost monitoring, and warn owners of webhook consumers to expect a burst when queued logs arrive.

The gateway is failing requests. Confirm the failure is Helicone's (no provider incident, errors originate at the gateway). Open the breaker in the shared client. Watch provider spend directly, because edge rate limits are now off. Close the breaker gradually once the gateway is healthy, and reconcile the unlogged window from your own application logs.

A Helicone API key leaked. Rotate with the two-key overlap immediately, but compressed: new key out, old key deleted within minutes rather than days. Because every key has write access, also check for unexpected spend on your provider keys through the gateway during the exposure window, and for unexpected score, feedback or property writes. If the key could read bodies, treat the logged prompts as disclosed and involve whoever handles data incidents.

Cost spike. Group by property first (Team, Tenant, feature), then by user, then by model. A single user or session dominating is a runaway loop or abuse; apply a u=cents limit scoped to user. A whole team rising is a deploy; find it. A model change is usually routing: an automatic model string (gpt-4o-mini with no suffix) routes cheapest-first across providers, and a newly configured BYOK key can change which provider serves traffic.

Sudden 429 "Insufficient credits". Since August 2026, Pass-Through Billing needs the ptb_enabled flag, and a failed flag lookup also denies it. Any service that relied on credits without BYOK keys breaks. Add provider keys in Provider Settings.

Tenant data seen by another tenant. Suspect the cache first: shared cache without a per-tenant seed. Disable caching for the affected route, purge by changing seeds, and add the seed in the shared client. Then check webhook filters and any property-based routing for the same class of mistake.

Self-hosted disk full. Bodies stop landing in MinIO or ClickHouse stops accepting inserts. Free space, then set retention: turn on body omission for high-volume workloads, and plan a size-appropriate volume or the Helm deployment on managed storage.

Where Helicone stops

Naming the boundary is part of owning a tool. Helicone is a strong fit for a particular job, and its maintenance-mode status sharpens the edges.

When you do pick something else, match the tool to the part of Helicone you are replacing. For tracing, sessions, prompts and evaluation, Langfuse is the most direct analogue and can be self-hosted in your own region. For a self-hosted, OpenAI-compatible gateway with routing, budgets and key management, the LiteLLM proxy fills the gap Helicone's self-hosted image leaves. For evaluation depth, the evaluation tools in this catalogue such as RAGAS cover what removed Experiments used to. Helicone's docs list integrations with several of these, which makes a gradual transition possible. None of this is an endorsement; it is a map of which capability goes where.

Running Helicone as a platform for other teams

Every Helicone feature is switched on by an HTTP header the caller sends. That is elegant for one team and a governance problem for twenty: a limit, a seed, a screening flag or a tenant property that a team forgets is simply absent, with no error. The central design move for a platform owner is therefore to stop teams setting Helicone headers themselves, and give them a shared client instead.

Ship a client, not a wiki page. A small internal library (one per language you support) takes your organisation's concepts (team, tenant, feature, user, session) and turns them into Helicone headers, with defaults the platform decides: a per-tenant cache seed whenever caching is on, a cost-based rate limit per user, stream_options.include_usage on streams, a team property on every call. Teams adopt it by changing an import. It is also where the break-glass fallback lives, and the one module you rewrite if you leave Helicone.

Govern the property taxonomy. Properties are free-form, so without a contract you get Env, env, Environment and environment in the same organisation, and every report needs four filters. Publish a short, fixed list (Team, Tenant, Feature, Environment) with allowed values, enforce it in the client, and treat new properties as a reviewed change.

Decide organisations versus properties once. Use separate Helicone organisations where the boundary must be hard: production versus non-production, and any tenant whose data a subset of staff may not read. Use properties for everything else. Record the decision so it is not relitigated per team.

Own alerts and showback centrally. A standard set of alerts per team (error rate, cost over a window grouped by Team, p95 latency) is created by the platform when a team onboards. A monthly HQL showback report goes to each team lead with the reconciliation against provider invoices. Numbers that arrive on a schedule change behaviour; a dashboard nobody opens does not.

Offer an onboarding path with a definition of done. A team is onboarded when its traffic arrives with the standard properties, its alerts exist, its break-glass path has been exercised once, and its owner knows the degraded-mode behaviour. Anything less is a team that will find out about the May 2026 shape during an incident.

Communicate the platform's status honestly. Tell teams Helicone is in maintenance mode, what that means (fixes yes, features no), which documentation pages are stale, and what the exit trigger is. Teams make better decisions when they know the platform's horizon.

The review checklist

Run this against any pull request that adds or changes an LLM integration, and against any design document that proposes Helicone for a new workload.

The complete picture

The series' final artefact: the shared client every team imports. It encodes the platform's decisions, and every line should be justifiable from an earlier section.

"""Shared LLM client: the only place Helicone headers are written."""
import os
import time
import uuid
from dataclasses import dataclass, field
from openai import OpenAI, APIConnectionError, InternalServerError

ALLOWED_TEAMS = {"search", "support", "risk"}          # the property taxonomy
ENVIRONMENT = os.environ.get("APP_ENV", "development")

_gateway = OpenAI(base_url="https://ai-gateway.helicone.ai",
                  api_key=os.environ["HELICONE_API_KEY"])     # service-owner key
_direct = OpenAI(api_key=os.environ["OPENAI_BREAKGLASS_KEY"])  # provider-capped key
_breaker_open_until = 0.0


@dataclass
class CallContext:
    team: str
    tenant: str
    feature: str
    user_id: str
    session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    session_path: str = "/"
    session_name: str = "default"
    cache: bool = False
    store_bodies: bool = True
    screen: bool = False

    def headers(self) -> dict[str, str]:
        if self.team not in ALLOWED_TEAMS:
            raise ValueError(f"unknown team {self.team!r}; add it to the taxonomy")
        h = {
            "Helicone-User-Id": self.user_id,
            "Helicone-Property-Team": self.team,
            "Helicone-Property-Tenant": self.tenant,
            "Helicone-Property-Feature": self.feature,
            "Helicone-Property-Environment": ENVIRONMENT,
            "Helicone-Session-Id": self.session_id,
            "Helicone-Session-Path": self.session_path,
            "Helicone-Session-Name": self.session_name,
            # $5 per hour per user, enforced at the edge
            "Helicone-RateLimit-Policy": "500;w=3600;u=cents;s=user",
        }
        if self.cache:
            h["Helicone-Cache-Enabled"] = "true"
            h["Helicone-Cache-Seed"] = f"{self.tenant}:{self.user_id}"
        if not self.store_bodies:
            h["Helicone-Omit-Request"] = "true"
            h["Helicone-Omit-Response"] = "true"
        if self.screen:
            h["Helicone-LLM-Security-Enabled"] = "true"   # OpenAI models only
        return h


def chat(ctx: CallContext, messages, *, model="gpt-4o-mini", stream=False):
    global _breaker_open_until
    extra = {"stream_options": {"include_usage": True}} if stream else {}
    if time.monotonic() >= _breaker_open_until:
        try:
            return _gateway.chat.completions.create(
                model=model, messages=messages, stream=stream,
                extra_headers=ctx.headers(), **extra)
        except (APIConnectionError, InternalServerError):
            _breaker_open_until = time.monotonic() + 60
            print("platform_llm: break-glass active, Helicone bypassed")
    return _direct.chat.completions.create(
        model=model.split("/")[0], messages=messages, stream=stream, **extra)

from platform_llm.client import CallContext, chat

ctx = CallContext(team="support", tenant="acme", feature="reply-draft",
                  user_id="u-812", session_name="Support ticket",
                  session_path="/ticket/draft", cache=True, screen=True)
reply = chat(ctx, [{"role": "user", "content": "Summarise ticket 4411"}])

Around that client sit the other pieces this guide built: the signed webhook receiver, the monthly HQL showback with the 1e9 divisor, the standard alerts per team, a monthly export job to storage you own, and, for self-hosters, a digest-pinned deployment with tested backups.

Where the series leaves you

Across three levels you have gone from a first logged request to owning Helicone as a platform with a known status and a planned horizon.

You should now be able to look at any LLM integration and see not just what it logs but what it risks: whether Helicone is on the critical path and what happens when it fails, who holds which key, whether one tenant can read another's cached answer, whether a rate limit or screen applies or only appears to, where the data physically lives, whether the self-hosted build contains the fixes that matter, and what leaving would cost.

Carry that review instinct forward. With an observability platform, as with containers and pipelines, most incidents are prevented at review time, and most migrations are cheap only if someone designed for them before they were needed.

Sources

• Helicone documentation index (llms.txt)
• Platform overview
• AI Gateway overview
• Provider routing
• Error handling and fallback
• Header directory
• Caching
• Custom rate limits
• Custom properties
• LLM security
• Moderations
• Omit logs
• Sessions
• HQL
• Webhooks
• Alerts
• OpenLLMetry async integration
• Gateway fallbacks (legacy)
• MCP server
• Self-hosting overview
• Self-hosting with Docker
• Self-hosting: manual setup
• Self-hosting: cloud deployment
• Self-hosting on Kubernetes
• Proxy vs async
• Availability
• Latency impact
• How cost is calculated
• Data autonomy
• Streaming usage
• Helicone is joining Mintlify
• AWS account incident write-up
• Pricing
• Helicone monorepo on GitHub
• Helicone Helm charts (v3)
• Standalone AI Gateway repository
• helicone-all-in-one on Docker Hub

Part three of three. A senior interview about Helicone is rarely about headers. It is about judgement: where an observability gateway belongs in an architecture, what happens when it fails, who holds which secret, and whether you would choose this tool at all in 2026. This file gives you system-design prompts with worked answers, incident-response questions, the leadership trade-off conversations, and a closing set of rapid questions. About forty-five minutes.

The thirty-second senior answer

Helicone is an open-source LLM observability platform with a gateway. Traffic either flows through its edge proxy or AI Gateway, which adds caching, rate limits, failover and screening by per-request headers and logs asynchronously after the response, or it is logged out of band with no edge features. Since March 2026 it has been in maintenance mode after joining Mintlify: fixes ship, features do not, new cloud sign-ups are closed, and the self-hosted image dates from August 2025. I would keep it where it already works, bound the risk, and have a written exit plan.

Saying the status unprompted is the senior signal. A candidate who describes Helicone as if it were growing and open for sign-ups tells the interviewer they have not looked recently.

The model to have ready

Four facts carry most senior answers. First, only the edge is on the critical path; logging happens after the response is returned, and feature code falls back to plain proxying if it fails. Second, every feature is opt-in per request by header, so anything a caller forgets simply does not happen. Third, all Helicone keys have write access (since June 2024), and member-created keys die with the member's organisation membership (since September 2026). Fourth, cost is stored multiplied by 1e9 in ClickHouse, so HQL reports divide by a billion.

System-design prompts

"Design LLM observability for thirty product teams on Helicone"

A weak answer lists features: sessions, properties, dashboards. A strong answer starts with the governance problem that per-request headers create and designs around it.

Start with the shared client. Teams do not write Helicone headers; they import an internal library that takes organisational concepts (team, tenant, feature, user, session) and emits headers with platform-chosen defaults: a fixed property taxonomy, per-tenant cache seeds whenever caching is on, a cost-based rate limit per user, usage reporting on streams, and a client-side circuit breaker that calls providers directly when the gateway is unreachable. That one module is the enforcement point, the documentation, and the migration seam if you ever leave.

Then decide integration method per workload. The rule: a workload goes through the gateway only if it uses an edge feature (cache, limits, failover, screening). Batch jobs and internal tools that use none of them use async logging, so they carry no availability risk from Helicone at all.

Then decide boundaries. Properties separate teams and tenants inside one organisation; separate organisations exist only where the boundary must be hard, such as production versus non-production, or data some staff may not read. Organisation counts are plan-limited, so this is a real constraint.

Then operations: a standard alert set per team created at onboarding (cost grouped by team, error rate, p95 latency), a monthly HQL showback reconciled against provider invoices, a monthly export to storage you own because retention is finite, and a degraded-mode table stating what stops when logging is down. Close by naming the status: in maintenance mode, you design so that leaving means rewriting the shared client, not thirty codebases.

"We are a Saudi bank. Can we use Helicone?"

This tests whether you can separate a technical answer from a compliance one without pretending to be a lawyer.

The facts: Helicone Cloud runs in US and EU regions only, and the response cache lives in Cloudflare Workers KV on Helicone's account. Helicone-Omit-Request and Helicone-Omit-Response prevent storage but bodies still transit Helicone's backend in memory. So if the bank's obligations require that prompts and completions do not leave the Kingdom, Helicone Cloud cannot meet them, and no header changes that.

The workable design is self-hosting in an in-country cloud region, with async logging, because the self-hosted all-in-one image does not proxy and there is no maintained self-hosted gateway. You give up caching, edge rate limits and gateway failover, and you take on the self-hosted supply-chain problem: the published image predates the 2026 security fixes, so you either build from main and own the release process, or restrict the install to a private network of trusted operators. A strong answer ends with the honest comparison: for a new regulated deployment, a tool with an actively maintained self-hosted distribution, such as Langfuse for tracing or a LiteLLM proxy for gateway functions, is likely the better starting point, and the decision should go to the bank's compliance team with that trade-off written out.

"Design a gateway layer so an LLM provider outage does not take us down"

The candidate who says "use Helicone's failover" is half right. The AI Gateway's automatic model string (gpt-4o-mini with no suffix) routes across every provider offering the model, cheapest first, with failover on 429, 401, 400 context-length errors, 408 and 5xx. A manual chain such as gpt-4o-mini/azure,gpt-4o-mini/openai,gpt-4o-mini sets the order explicitly. That handles a provider outage.

The other half is that the gateway itself is now a single dependency. So the design has two layers: provider failover inside Helicone, and a client-side breaker around Helicone that falls back to a direct provider call with a separately held, spending-capped key. The breaker must translate Helicone's model grammar (the /openai suffix, !provider exclusions) into a plain provider model name, must alert while open, and must be exercised in game days, because a fallback that has never run does not work. Mention what you lose in break-glass mode (logging, limits, cache) and how you reconcile the gap afterwards from application logs.

"Make our LLM spend explainable to finance"

Correct measurement first, then attribution, then control. Measurement: streams must send stream_options.include_usage or helicone-stream-usage: "true", and unresolvable model names need Helicone-Model-Override, or the numbers are silently wrong. Attribution: a mandatory Team and Feature property from the shared client, and an HQL showback grouped by them, remembering the 1e9 divisor. Reconcile monthly against provider invoices and explain any gap. Control: cost-based edge limits (u=cents, scoped per user or per property) for fast protection, cost alerts grouped by team for awareness, and provider-side spending caps as a floor Helicone cannot bypass. Note that alerts depend on the logging path, which was down for about four days in May 2026, which is why the provider caps exist.

Incident-response questions

"The Helicone dashboard has shown no new requests for two hours. What do you do?"

First, separate traffic from telemetry: are users failing? Check application error rates and look for helicone-id headers on responses. If traffic is healthy, this is the logging path, the same shape as the May 2026 AWS account incident, when ingestion and the dashboard were down for about four days while the proxy stayed up and logs were queued without loss. Do not open the break-glass breaker: the edge is working and its limits still protect you. Tell finance and teams that alerts are blind, switch cost monitoring to provider usage reports, and warn webhook consumers to expect a burst. Afterwards, reconcile, and check that the degraded-mode table matched what happened.

"Customers report seeing someone else's answer."

Suspect the cache first. The cache key includes Authorization, which on the AI Gateway is your single Helicone key for all tenants, so identical requests from different tenants share entries unless a seed separates them. Disable caching on the affected route immediately, change seeds to invalidate existing entries, add a per-tenant seed in the shared client, and treat it as a data incident: identify which responses could have been served across tenants in the window. Then check the same class of mistake elsewhere: webhook property filters, property-scoped rate limits, anything keyed on an optional header.

"Every service got 401s this morning. Nothing was deployed."

Ask who left the organisation yesterday. Since September 2026, an API key created by a non-owner member carries that member's user ID and fails membership checks once they leave. Create new keys from a service-owner account, roll them out, and then inventory every remaining key by creator. The prevention is a policy that production keys are only ever created by the platform account.

"Self-hosted Helicone: a security researcher says our dashboard is vulnerable to stored XSS."

They are probably right if you run v2025.08.21: the fixes that strip dangerous HTML from LLM output and add DOMPurify sanitising landed in early 2026 and exist only on main. Immediately restrict network access to the dashboard to trusted operators. Then decide between building from main (pin a commit, build in your CI, back up Postgres, ClickHouse and MinIO, run migrations, validate the new proxy-key constraint, canary) and retiring the self-hosted install. Also review the other 2026 fixes you lack: unauthenticated SQL endpoints, HQL tenancy, admin takeover.

"The gateway started returning 429 'Insufficient credits'."

Pass-Through Billing has required the ptb_enabled organisation flag since August 2026, and a failed flag lookup also denies it. Any service relying on credits without provider keys configured breaks. Add BYOK keys in Provider Settings. In a post-incident note, flag that the documentation's headline flow no longer applies to most organisations.

Arguing trade-offs to leadership

Leadership does not want the header list. They want the decision, its cost, and its risk, in terms they already use.

"Should we keep Helicone?" Frame it as run, bound and plan. Run: it works today, the cloud receives security fixes, and switching has a real cost. Bound: move non-edge workloads to async logging, add a break-glass path, put production keys on a service account, export history monthly. Plan: a shared client makes the exit a module rewrite, an asset inventory prices it, and a named trigger (an end-of-life date, an unshipped security fix, a residency requirement) starts it. That answer shows you are neither attached to the tool nor alarmist about it.

"Why not just call providers directly and skip the gateway?" Show what the gateway buys in the currency leadership cares about: cache hits as dollars saved, cost limits as incidents prevented, failover as minutes of avoided downtime. Then show the cost: a vendor on the request path, a small latency hop, and a dependency in maintenance mode. If the benefit column is empty for a workload, agree with them for that workload.

"Can't we self-host to save money and keep data here?" Yes for data, with conditions. The self-hosted image lacks the gateway and the 2026 security fixes, so self-hosting means either an engineering commitment to building and upgrading from main, or accepting a private-network-only install with async logging. Put a number on the engineering time, compare it with the vendor bill, and include the alternative of a different, actively maintained self-hosted tool.

What a strong answer covers

Across every question above, interviewers listen for the same five things, and weak answers miss them in the same ways.

Common interview questions

1. What is Helicone's status today?In maintenance mode since joining Mintlify in March 2026. Security fixes, new models and bug fixes continue in the cloud; new sign-ups are disabled; self-serve upgrades were removed; the newest self-hosted image is v2025.08.21; the standalone Rust gateway is abandoned.
2. What is on the critical path?Only the edge: the Cloudflare Workers proxy or AI Gateway. Logs go to Kafka after the full response is returned, then to Jawn, ClickHouse, Postgres and object storage. Feature failures fall back to plain proxying.
3. When would you not route through the gateway?When the workload uses no edge feature. Then async logging gives the same observability without adding Helicone to the request's success path.
4. What can a leaked Helicone key do?Everything a key can: send traffic billed to your BYOK provider keys, read logged bodies through the API, and write scores, feedback and properties. All keys have had write access since June 2024.
5. Why might keys stop working when someone leaves?Since September 2026, keys created by non-owner members carry the member's user ID and fail membership checks after they leave. Create production keys from a service-owner account.
6. How can caching leak data between tenants?The cache key includes Authorization, which on the gateway is one shared Helicone key. Identical requests from different tenants share entries unless Helicone-Cache-Seed separates them.
7. Does Helicone-Omit-Request keep data away from Helicone?No. It prevents storage; bodies still transit the backend in memory. For data that must not leave, use async logging with content tracing disabled.
8. What does LLM security screening cover?Prompt Guard for jailbreaks and injection, plus Llama Guard when advanced mode is on, for OpenAI models only and only on requests carrying the header.
9. How do you make rate limits unavoidable?You cannot at Helicone's level, because the policy is a request header. Enforce it in a shared client, and keep provider-side caps as a floor.
10. What changed in HQL in September 2026?The validator rejects quoted identifiers, SET/SETTINGS, heredocs, # comments, multiple statements, database-qualified names and table functions; FROM must be request_response_rmt or a subquery.
11. How do you keep a self-hosted install secure?Decide explicitly: build from main with pinned commits, backups and migrations, or restrict the published image to a private network. Change BETTER_AUTH_SECRET and the MinIO defaults, firewall 5432 and 8123, and put TLS in front.
12. What breaks first on a self-hosted node?Disk, usually object storage full of bodies, then ClickHouse. Body omission for high-volume workloads is the biggest lever.
13. How do you plan an exit?Export history with @helicone/export, swap the shared client's destination behind a flag with a dual-logging period, and recreate Helicone-only assets: prompts, saved queries, alerts, webhooks.
14. How do you verify webhooks?HMAC-SHA256 over JSON.stringify of the body (hex in the docs' style of example, confirm against a real delivery), compared in constant time against helicone-signature after a length check. Acknowledge fast: delivery times out at two minutes, and the full-body URL expires at thirty.

Final self-test

• State Helicone's status and three concrete consequences for an owner.
• Draw the request path and mark which components are on the critical path.
• Give the rule for choosing gateway versus async per workload.
• Describe the client-side breaker and two ways it fails if never exercised.
• Name the five parties in the trust model and what each can do if compromised.
• Explain the cross-tenant cache trap and its fix.
• Explain why a departing engineer can cause an outage, and the prevention.
• Say what the self-hosted v2025.08.21 image lacks and the two ways to handle it.
• Give the data-residency pattern for a Gulf or Egyptian employer.
• Write an HQL showback query from memory, including the cost divisor.
• Walk the "dashboard is empty" incident without opening the breaker, and say why.
• Frame "should we keep Helicone?" for leadership in three words and three sentences.

Part three of three. At this level the problems are not failed requests but quiet ones: a rate limit that never applied, a cache that answered the wrong customer, a key that died with an offboarding, a self-hosted install a year behind on security fixes. Start with the error table, then the traps that only appear at scale, the habits that prevent them, and the checklists for pull requests and design documents.

Common errors at this level

Everything from the beginner and mid levels still applies. These are the ones that cause incidents rather than failed requests.

Production traps that only appear at scale

These do not show up in a proof of concept with one team and one tenant. They appear when the platform has many callers, many tenants, and months of history.

Optional headers decay. At one team, every feature header is set deliberately. At twenty, some code paths are written by people who never read the platform docs, and every header they omit is a feature silently absent: no limit, no seed, no screening, no team property. Nothing errors. The only defence is to take header-writing away from callers and put it in a shared client, then measure compliance with an HQL query that counts requests missing the standard properties.

The cache becomes a data store. With one tenant, the cache is an optimisation. With many, it is a shared store keyed on request bytes, and templated prompts make byte-identical requests common. Once any cached answer contains tenant-specific content, cross-tenant hits are disclosures. Per-tenant seeds are not a tuning option at scale; they are part of the isolation model.

Retention turns into a history problem. Three months of data on the Team plan feels generous until someone asks for year-on-year cost per feature, or a regulator asks what the model said to a customer last spring. By then the data is gone. Monthly exports to storage you own are cheap only if they started before you needed them.

Alerts share the failure domain they monitor. Cost and error alerts are computed from logged data. When logging stops, as it did for about four days in May 2026, alerts stop too, exactly when you most need to know whether anything else is wrong. At scale, keep an independent floor: provider-side spending caps and application-side error metrics that do not depend on Helicone.

Key sprawl outlives the people who created it. A year in, dozens of keys exist, many created by engineers who have since moved teams or left. The September 2026 change turns that sprawl into outages. Inventory keys by creator, and make key creation a platform-owned action.

Self-hosted drift compounds. Each month on the August 2025 image adds to the list of fixes you lack and to the diff you must read to catch up. A self-hosted install that is not upgraded on a schedule becomes one that cannot be upgraded safely at all.

Routing changes cost without a deploy. An automatic model string routes cheapest-first across every provider with a configured key. Adding a BYOK key for a new provider, which feels like an administrative change, can move traffic to a different provider with different latency and behaviour. Treat Provider Settings changes as production changes with review.

The practices that pay off most

In order of leverage.

Platform-ownership habits

Read the repository weekly. The 2026 changes that mattered (removed routes, the HQL validator, key semantics, the PTB flag) arrived as merged pull requests on Helicone/helicone, not as release notes. Fifteen minutes a week skimming merged pull requests is the cheapest early-warning system you have.

Keep a degraded-mode table. One row per Helicone function your teams use: what stops, who notices, and the manual workaround. Review it after every incident, yours or Helicone's.

Exercise the break glass. Once a quarter, force the client-side breaker open in staging and confirm requests succeed directly, the alert fires, and the model-string translation works. Once a year, do it in production for a few minutes with the teams warned.

Publish the status honestly. Tell teams Helicone is in maintenance mode, which documentation pages are stale (the "sign up for free" quickstart, the Experiments cookbook, the self-hosted port table's "LLM Proxy" label), and what the exit trigger is. Teams that know the horizon stop building deep dependencies on Helicone-only features.

Price the exit once a quarter. Update the inventory of Helicone-only assets: prompts in {{hc:...}} syntax, saved HQL queries, alerts, webhooks, datasets. The number should go down or stay flat. If it grows, someone is building lock-in, and you want to know before they finish.

Treat Provider Settings as production config. Adding or rotating a BYOK key can change routing and therefore cost and behaviour. Changes go through the same review and announcement as a deploy.

Measure compliance, not intent. A weekly HQL query that counts requests missing the Team property, or cached requests without a seed, tells you how well the shared client has been adopted far better than any survey.

SELECT
  count()                                   AS total,
  countIf(properties['Team'] = '')          AS missing_team,
  round(missing_team / total * 100, 1)      AS pct_missing
FROM request_response_rmt
WHERE request_created_at >= now() - INTERVAL 7 DAY

Practice cards

1. Leak a cached answer, then stop itThrough the gateway with caching on and no seed, send one request as tenant A and the same as tenant B. Read Helicone-Cache on the second. Add a per-tenant seed and repeat.
2. Watch a limit not applySet s=tenant with a tiny quota, send requests without the tenant property, and count how many succeed. Then add the property and find the 429.
3. Break the glassPoint your client's gateway URL at an unroutable address and confirm the breaker opens, requests succeed directly, and a log line or alert says so.
4. Reconcile a monthRun the showback query with the 1e9 divisor and compare with your provider invoice. Name the cause of the gap.
5. Trip the HQL validatorWrite a query with a double-quoted identifier, then one with numbers(), and read both errors so you can recognise them in a team's bug report.
6. Kill a key by offboardingIn a test organisation, have a non-owner member create a key, remove the member, and watch the key fail.
7. Fill a self-hosted diskLog a thousand realistic requests to the all-in-one image with volumes mounted and measure which volume grows fastest. Repeat with bodies omitted.
8. Export and resumeRun @helicone/export for one month, interrupt it, and resume from the checkpoint.

Verifying, not assuming

Each of these confirms a control is live rather than configured.

# Is the response coming through Helicone, and was it cached?
curl -si https://ai-gateway.helicone.ai/chat/completions \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Helicone-Cache-Enabled: true" \
  -H "Helicone-Cache-Seed: tenant-a" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  | grep -iE '^(helicone-id|helicone-cache)'

# Is the rate limit live? Look for the limit and remaining headers.
curl -si https://ai-gateway.helicone.ai/chat/completions \
  -H "Authorization: Bearer $HELICONE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Helicone-User-Id: u-1" \
  -H "Helicone-RateLimit-Policy: 10;w=60;s=user" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  | grep -i '^helicone-ratelimit'

# Self-hosted: is Jawn healthy, and is the image pinned?
curl -s http://localhost:8585/healthcheck
docker inspect helicone --format '{{.Image}}'

# Self-hosted: does the browser config point at the public Jawn URL?
curl -s http://YOUR_IP:3000/__ENV.js | grep JAWN

A security control you have never watched refuse anything is decoration. For every row above, also run the negative case: the wrong seed, a missing user header, an unpinned tag.

Incident quick reference

Dashboard empty, users fine: logging path. Do not break glass. Announce blind alerts, watch provider usage, expect a burst later.

Cross-tenant answer: disable caching on the route, rotate seeds, add per-tenant seeds in the client, treat as a data incident.

Mass 401s: check recent offboardings first; recreate keys from the service account.

Spend spike: group by team, then user or session, then model. A single session is a loop; a whole team is a deploy; a model shift is routing.

Leaked key: compressed two-key rotation, check provider spend and unexpected score or property writes during the window, treat logged bodies as disclosed.

Agent workloads specifically

Agents are where Helicone's per-request model is stretched hardest, because one user action becomes dozens of LLM calls, tool calls and retries, and a bug becomes a loop.

Scope cost limits to the session's owner, not the request. A per-request rate limit does nothing to an agent that makes many cheap calls. A cost-based policy scoped per user (u=cents;s=user) bounds what one user's agent runs can spend in a window, which is the failure you actually fear: an agent stuck in a tool loop at three in the morning. Remember the minimum window is 60 seconds and only one policy applies per request, so choose the window that matches how fast a loop burns money.

Make session paths describe function, not time. Helicone-Session-Path builds the trace tree. Paths like /plan, /plan/search, /plan/search/summarise let you compare the cost of each step across thousands of runs. Paths built from timestamps or step counters produce a unique tree per run and make aggregate analysis impossible. Put the path convention in the shared client, not in each agent.

Beware retries multiplying under failover. Helicone-Retry-Enabled retries 429 and 5xx up to five times by default, and each attempt is logged separately. Combined with gateway provider failover and your agent framework's own retries, one failing step can become many billed calls. Pick one retry layer per workload and turn the others off; the docs themselves suggest gateway failover is usually better than retries unless you are pinned to one provider.

Log tool spans too. Tool and vector-database calls can be logged into the same session with the logger SDK, so the trace shows where an agent spent its time rather than only its model calls. Without them, a slow retrieval step looks like a slow model.

Self-hosted operations at a glance

If you run the self-hosted stack, a few routines carry most of the operational load.

Back up all three stores together. Postgres holds organisations, keys and prompts; ClickHouse holds metrics; object storage holds bodies. A restore of one without the others leaves requests without bodies or keys that point nowhere. Snapshot them in the same window and practise the restore on a scratch host at least once per upgrade.

Keep one origin. SITE_URL, BETTER_AUTH_URL and NEXT_PUBLIC_APP_URL must match, and NEXT_PUBLIC_HELICONE_JAWN_SERVICE and S3_ENDPOINT must be reachable from browsers, not only from the server. Mixing localhost with a public address gives "Invalid origin" at sign-in; a missing NEXT_PUBLIC_IS_ON_PREM=true gives a redirect loop. These look like authentication bugs and are configuration.

Put an upgrade on the calendar. Pick a cadence (quarterly is realistic), and at each one decide explicitly: move to a newer main commit after reading the merged pull requests, or record why you are staying. The worst state is a self-hosted install nobody has decided anything about.

Review checklist for pull requests

Review checklist for design documents

The senior habit behind both tables is the same: read an integration for what it permits and what it assumes, not only for what it does. Most Helicone incidents at scale are a header someone assumed was set, a boundary someone assumed was there, or a platform someone assumed would keep growing.