The Real Cost of Running an AI Feature in Production

Most teams estimate the cost of an AI feature by opening a provider pricing page, multiplying tokens by requests, and calling it a budget. Then the feature ships, real users arrive, and the number on the invoice looks nothing like the spreadsheet. The gap is not a rounding error. It is the difference between the price of a single API call and the cost of running that call reliably, at scale, for people who expect it to work every time.

We build and operate AI features for a living, so we tend to look at the whole ledger before anyone writes code. Here is what actually sits inside the cost of AI in production, and where the surprises hide.

The token bill is the part everyone sees

Per-request inference cost is real, and it is the easiest line item to model. You count input tokens, output tokens, and volume. It is also the line item people over-index on, because it is the only one the provider hands you neatly.

Two things distort it once you leave the demo. First, prompts grow. System instructions, retrieved context, few-shot examples, and conversation history all ride on every call, and they rarely shrink over time. Second, real traffic is spiky. You size for the average and then pay for the peak, because the peak is when your users are actually watching.

If your estimate stopped at tokens, you have priced the ingredient, not the meal.

Retries, failures, and the cost of "usually works"

Models time out. Providers rate-limit you. A response comes back malformed and fails your schema check. In production, every one of these is a retry, and every retry is another paid call plus latency the user feels.

A feature that works ninety-five percent of the time on the first try sounds fine until you notice the other five percent is costing you double calls, fallback logic, and a support ticket. Building for the failure path is not optional engineering polish. It is a cost centre you either budget for deliberately or absorb by surprise.

The infrastructure around the model

The model is a single component in a system that has to stand up on its own. That system usually includes:

• A backend to hold your keys, enforce rate limits, and keep secrets off the client
• A queue or worker layer for anything that cannot finish inside a request cycle
• Caching, so you stop paying to answer the same question twice
• A datastore for logs, embeddings, or conversation state
• Egress, storage, and the compute that runs all of the above

None of this appears on the model provider's bill. All of it appears on yours. For voice and real-time features the picture is heavier still, because you are now paying for streaming transport, speech-to-text, text-to-speech, and telephony on top of the language model. When we built Voxif as a real-time voice platform, the language model was one line in a stack that also had to carry audio in and out with low latency, reliably, on the customer's own carrier.

Observability is not free, and skipping it is expensive

You cannot manage what you cannot measure, and AI features are unusually hard to measure. Output is non-deterministic, quality is subjective, and a regression can be a subtle drift in tone rather than a stack trace.

That means real logging, real evaluation, and real alerting: capturing prompts and responses, scoring quality over time, and catching the day a provider quietly changes a model behind the same name. This tooling costs money to run and time to maintain. Teams that skip it do not save the cost. They defer it until a silent failure reaches a customer, which is the most expensive place to discover it.

The people cost nobody puts in the spreadsheet

An AI feature is not "done" at launch the way a static page is. Prompts need tuning. Providers ship changes. Edge cases surface that no one imagined. Someone has to watch the dashboards, read the failure logs, and decide when a model update means it is time to re-test everything.

This ongoing engineering attention is usually the largest real cost of AI in production, and it is almost always missing from the original estimate. If your plan assumes zero maintenance, your plan assumes a feature that does not exist yet.

How to estimate it honestly

You do not need perfect numbers. You need a complete list. Before committing, price the full picture:

• Inference at peak volume, with realistic prompt sizes, not demo prompts
• A retry and fallback budget for the failure path
• The supporting infrastructure the model depends on
• Observability and evaluation tooling
• The recurring human time to keep it healthy

We publish the measurements rather than shipping on vibes, and this is one of the measurements worth publishing internally before you build. An AI feature can absolutely be worth it. But "worth it" is a judgement you can only make once the cost you are comparing against is the real one.