What an AI gateway does, and when you need one

The second AI feature is where the problem appears. The first one ships with a provider SDK, an API key in a config file, and a prompt built inline. That works. Then a second team builds something similar, picks a different model, and stores the key somewhere else. By the fourth, nobody can answer which systems call which provider, what they send, or what it costs.

A gateway is the layer that prevents that. Applications call one internal interface. What happens behind it becomes an operational decision rather than a code change in four repositories.

What belongs in the layer

• Routing: which model serves a given request, decided by policy rather than by whichever SDK a team imported
• Authentication and quota: which internal system is calling, what it may reach, and how much of it
• Logging: the request, the model and version, the context supplied, and the response, retained for as long as your obligations require
• Redaction: removing what must not leave the boundary before the call is made, not after
• Fallback: what happens when a provider is rate limiting, degraded, or gone

Each of these is something teams otherwise implement separately, inconsistently, and usually only after an incident makes it necessary.

What does not belong in it

Gateways decay when they accumulate application logic. Prompt templates, retrieval, business rules, and evaluation belong with the application that owns the use case. A gateway that knows what a customer service answer should look like has become a shared dependency that every team has to negotiate with before shipping.

Keep the boundary at transport and policy. If a change to a prompt requires a gateway release, the boundary is in the wrong place.

When it is worth building

Not for a single application. One system with one provider does not need an abstraction layer, and adding one early buys indirection nobody uses.

The threshold is the second production workload, or the first one that touches material with residency or contractual constraints. At that point the questions arrive quickly: what are we sending to whom, can we prove it, and what happens when we have to move. A gateway answers all three in one place.

The cost of adding a gateway is a week. The cost of adding one after six applications have a provider SDK compiled into them is a migration.

The audit argument

Most organizations discover the need through an audit question rather than an architecture review. Which systems process personal data through a model. What was sent on a particular date. Who authorised that system to use that provider.

Without a gateway, answering means asking every team and trusting the replies. With one, it is a query. That difference is worth more than the routing flexibility that usually gets top billing in the business case.

What we recommend

Build it when the second production workload appears, and keep it thin. Transport, policy, logging, and nothing that knows about your domain. Use an existing gateway product if one fits your control requirements, since this is not a differentiating component and the interesting work is elsewhere.

The test of a good one is that moving a workload from a hosted provider to your own infrastructure changes configuration and nothing else.

Whether a gateway is worth building comes down to one honest count: how many places in the organisation already call a model directly, and how many more will in the next year. That count is the first thing we establish in an operational AI engagement, because it decides whether the gateway pays for itself before the second workload or the fifth.

What is an AI gateway?

A layer between your applications and whichever model serves them. Applications call one internal interface, and routing, authentication, quota, logging, redaction, and fallback are handled in that layer rather than repeated in each application.

When should an organization build one?

At the second production workload, or at the first one involving material with data residency or contractual constraints. A single application with a single provider does not need one, and building it earlier adds indirection with no user.

Should prompts and retrieval live in the gateway?

No. Those belong with the application that owns the use case. A gateway holding domain logic becomes a shared dependency every team must negotiate with, and a prompt change should never require a gateway release.

Do we need to build a gateway ourselves?

Usually not. Several products cover routing, logging, and quota adequately. Evaluate them against your control and residency requirements first. This is not a differentiating component, and the engineering effort is better spent on the retrieval and integration work around it.