The platform we run our own AI on

Each application arrives with its own idea of how it is deployed. One has a pipeline that applies manifests directly, one is a Helm command somebody keeps in their shell history, one was set up during an evening and works. Nothing is wrong until something has to change, and then the question of what is actually running has no answer that does not begin with logging into the cluster and looking.

Secrets are the sharpest version of the same problem. They are created out of band because they cannot go in Git in the clear, so the one thing that must never be lost is the one thing with no record. Rotating a credential means finding every namespace it was typed into.

Then the drift. Somebody patches a live resource to get past an incident at eleven at night, correctly, and the fix exists only in the cluster. The next reconciliation from the repository — or the next fresh deploy — quietly removes it. The environment and the description of the environment have separated, and nothing reports that they have.

For an inference tier this gets worse rather than better. Models are large, slow to load, and pinned to specific GPUs. A deployment that looks healthy the moment it starts can be thirty seconds away from serving anything, and traffic sent in the meantime fails in ways that read like a model problem rather than a rollout problem.

One repository on a single main branch holds three root Applications: shared, test and prod. Shared carries the platform every environment depends on — ingress, certificates, storage, the Postgres operator, the secrets machinery, monitoring, the inference stack. Test and prod each carry their own stateful services and their own product applications. Each root pulls in its children, so adding an application means committing one Application manifest, not running anything.

Every product application has one base and thin overlays per environment. The overlay patches only what genuinely differs: the namespace, the public host, the TLS secret name, and which Postgres service and credential to use. When the same manifests are duplicated per environment instead, the two copies diverge within weeks and the test environment stops predicting anything about production. A base with a four-field patch cannot drift that way.

Each environment has an AppProject that names the repositories it is allowed to deploy from, as an explicit list rather than a wildcard. It is a small piece of configuration that decides something large: a mistaken or compromised source cannot deploy into a trusted namespace just because it can reach the cluster.

Secrets are resolved rather than stored. Twenty-eight ExternalSecrets pull their values from Vault at runtime, so what lives in Git is the reference and the shape, never the value. Nine SealedSecrets remain from the scheme this replaced, and they still work, which is precisely why that migration is not finished.

One namespace holds the inference tier: a gateway that presents a single API, open-weight models served on local GPUs behind it, three embedding services, a reranker, a vector database, and the chat interface people actually use. Model deployments are one base with per-model overlays that patch what differs — which weights, which GPU, how much memory, how long to wait for a load. Adding a model is a small file, not a new deployment to maintain.

The part that matters for governance is a NetworkPolicy. Model pods accept connections from the gateway pods and from monitoring, and from nothing else. An application inside the cluster cannot reach a model directly even if someone configures it to, because there is no route. Gateway-only access stops being a rule people follow and becomes a property of the network, which is the difference between a policy and a diagram.

That single control is what makes the rest answerable. Because every request passes one place, routing, rate limits, keys and the record of what was asked have one place to live. Prompts and documents stay inside the cluster by default, and reaching an external provider is a routing decision made deliberately in one file rather than a library import made incidentally in twelve.

It also makes the ordinary economics work. The models are already paid for and the machines are already there, so inference that would otherwise be metered per token is instead a question of whether the GPUs are busy — which is what makes it reasonable to point them at work nobody would fund per-token, such as the deferred defect backlog they chew through overnight.

The machines are described, but not reconciled. A node begins as a golden image in the hypervisor, cloned and given its identity — hostname, address, keys — at first boot, deliberately carrying none of that in the image so no clone can inherit another node's identity. It then runs one bootstrap script that takes a role, joins the cluster and prepares its storage. That is a real procedure and it is written down. What it is not is a desired state: it runs once, imperatively, and nothing afterwards notices if a node drifts away from what it was set up to be. The cluster self-heals; the machines under it do not.

That layer also lives outside the repository this case is about, in an older one where it is not yet committed. So the platform has two halves with different disciplines — one reconciled continuously from version control, one scripted and held locally — and the honest description of the second is that it is documented rather than governed. Closing that gap is ongoing work, and this case cannot report it as done.

Two manifests are applied by hand — the load-balancer pieces that must exist before the reconciler has anything to reconcile through. Bootstrap has to start somewhere. They are kept in a single folder so that the exception can be counted and pointed at, rather than dissolving into general practice.

The network is not default-deny. Three NetworkPolicies exist: the model tier and the secrets ingress are fenced, and the rest of the cluster is internally open. Fencing what matters first is a reasonable order of work, but describing that as a segmented network would not be true.

Test and production are separate namespaces with separate databases on the same cluster and the same hardware. They are isolated from each other's mistakes, not from each other's failures. For a platform this size that is a defensible cost decision, and it is a cost decision rather than an architectural one.

The secrets migration is unfinished, and the reason is the ordinary one. Twenty-eight references resolve from Vault; nine sealed secrets from the previous approach remain, and they work. Nothing forces the last nine, so they are still there — which is worth stating plainly, since a platform case that reports only completed migrations is describing a plan rather than a system.