insight
Bugs fixed overnight, reviewed in the morning
Software architect and consultant. Works with business-led product development, distributed systems, operational AI, and production software delivery.
Answer first
We built an in-house service that works the backlog of small, deferred defects while the office is empty, using computing capacity the organisation has already paid for. What makes it usable is not the model. It is the narrow scope, the team's own tests, and the review queue waiting at 08:00. The more interesting property is that the loop from report to released fix can close — which makes autonomy a decision about which changes you trust, rather than an engineering leap.
Every software team keeps a list of faults that are understood, repeatable, and never scheduled. They are real enough to write down and small enough to lose. Each planning round reaches the same conclusion: something more valuable is happening this month.
We built an in-house service to work that list while the office is empty. It takes a bug report, works out which part of the software it belongs to, assesses and prioritises it, reads a limited part of the code, prepares a small change, and runs the team's own tests against it in a sealed, disposable copy of the system. The result goes into a review queue. In the morning a person approves it, and only then does anything reach the real code — as a proposed change, marked as a draft, entering exactly the review the team already uses for its own work.
No single step in that sequence is novel. What makes it useful is when it runs and what it is not permitted to do.
That combination is the unusual part. A good deal of software will now write code from a description. Much less of it is built to refuse, to show its evidence, and to run inside your own network on hardware you have already paid for. Those three properties are the difference between something an organisation can put into service and something it can only admire.
The scarce resource was never the fix. It was the attention required to prepare one.
The night shift nobody was using
When an organisation runs AI models on its own hardware, that hardware is sized for the daytime peak: the assistants, the search, the operational systems people use while they are at work. It is bought, powered and written down on that basis. Between the last person leaving and the first arriving, most of it does very little.
That is the reversal which makes overnight maintenance work interesting. Rented by the request, a long attempt — with second tries, with the tests run again and again — is a line on an invoice, so the instinct is to keep it short. On hardware you already own, the same attempt competes with nothing. What it costs is electricity and wear, and the alternative use of those hours is nothing at all.
One day of capacity
The window the day leaves behind
- 01Ceiling — provisioned for the daytime peak
- 0208:00–18:00 — interactive demand, latency matters
- 0318:00 — the load falls away
- 0423:00–07:00 — the deferred queue fills the headroom
A run that retries three times costs the same as one that does not.
The ceiling is paid for either way.
| Property of the work | Rented by the request | On hardware you own |
|---|---|---|
| An attempt that has to be retried three times | three times the bill | the same fixed cost |
| Nobody waiting for the result | no advantage | fills hours that were idle |
| Source code and fault reports as the input | a procurement and data-transfer question | the question does not arise |
| What limits how much gets done | the budget | the hardware, and the length of the night |
It is also the kind of work that suits your own hardware best. It runs in batches, nobody is waiting on it, a second attempt costs nothing, and it is made entirely of material many organisations will not send anywhere: source code, fault reports, test output, internal identifiers. Teams that have already chosen to run models on their own infrastructure for those reasons usually have the spare capacity and the constraint at the same time.
What happens between 23:00 and 07:00
A report arrives through a route that knows who sent it: an issue in whatever tracking tool the team already uses, or a feedback form in the product itself. It is matched to the right piece of software automatically when that is obvious, and by a person when it is not. The system then summarises the report, suggests which part of the product is affected, sets a priority, and looks for similar reports from the past. Where it is not confident, it stops and asks for a human view rather than proceeding on a guess.
worth knowingThe best place to report a problem is the product itself.
One report, one night
The pipeline and its approval point
- 01Intake
- 02Triage and priority
- 03Bounded fix
- 04Isolated tests
- 05Approval — a person decides
- 06Draft pull request
A proposal that fails verification stops at the tests and is held with a failure report.
Nothing crosses the approval point on its own.
Lower-priority work is scheduled for the quiet hours, and only a set number of jobs run at once, so the queue drains steadily rather than all at once. Each job reads only the part of the software it has been given, keeps its change under an agreed size, and then has to survive the team's own tests before it can be offered to anyone.
A real, reproducible defect, triaged and prioritised. It has now lost three planning rounds to work that mattered more, which was the correct decision each time.
Nothing here is mysterious. It is understood, small, and never the most valuable thing to do next.
Follow one deferred fault through a single night, then work the morning queue and decide what is allowed to reach the real code.
The controls are the product
The interesting engineering is not the part that writes the change. It is the boundary around it. A reviewer can only judge a proposal in minutes if they already know what it cannot possibly contain.
Inside the envelope
What a run is allowed to touch
- 01Configured source paths
- 02A change-size ceiling
- 03One fixed test command
- 04An isolated branch
Outside it: deployment manifests, CI configuration, credentials, lockfiles. Refused, not reviewed.
The authority is narrower than the capability.
- It may read and change only the areas of the software it has been given, and nowhere else.
- A limit on how much it may change, so a proposal that grows into a redesign is rejected rather than reviewed.
- Protected files — deployment settings, build configuration, credentials, anything sensitive — are refused outright.
- One fixed set of tests per system, chosen by an administrator, run in a sealed and disposable copy of it.
- Optionally, the team's own automated build has to pass before a change can be offered at all.
- Approval required by default. Publishing without it is a deliberate exception, set for one system at a time.
- A limit on how much runs at once, a schedule, a pause, and an emergency stop that halts anything leaving the system immediately.
Each control says the same thing differently: what the system is allowed to do is deliberately narrower than what it is able to do. That is also the honest answer to the trust problem with code written by AI — not better writing, but less that can go wrong, and evidence attached to every proposal.
The morning queue
A reviewer opens a night's work and sees the explanation, the change itself, which files it touched, what the tests said, any failure report, and what it cost to produce. They approve it, reject it, ask for another attempt, or cancel it. On approval the change is checked again, set aside on its own branch, and opened as a draft for the team to review — waiting first, if the team wants, for their automated build to pass.
Five runs, three outcomes
What one night produces
- 0101 · verified proposal — waiting for approval
- 0203 · held by the checks — failure report retained
- 0301 · refused by a control — protected path
Only the first is a fix. The other two are the system working.
Repository writes without a person: none.
Three outcomes matter. A checked proposal waiting for approval. Work the tests stopped, with the failure kept for someone to read, which means the tests did their job. Work a control refused, which means the boundary did its job. A night that produces one approved fix and four honest refusals is a good night; a night that produces five accepted changes nobody read would not be.
The system is not trusted to be right. It is trusted to stop.
Because every attempt records what it was given, what it read, what it proposed and which checks it ran, a rejected proposal is as useful as an approved one. It shows where the software is hard to work in, which tests cannot be relied on, and which reports were never specific enough to act on. That is the same argument as answer playback, applied to maintenance work rather than to a generated answer.
The loop can close
There is a second property here, and it is the more interesting one. Look at the steps between a report arriving and a fix being live: intake, assessment, the change itself, the tests, the review, the release. Every one of them is something the organisation has already automated, or already could. Exactly one is a person by deliberate choice.
One reported defect, one circuit
The loop a reported defect can travel
- 01Report — a person notices
- 02Triage — priority and component
- 03Bounded fix — inside the envelope
- 04Verify — tests and CI
- 05Approval — the one deliberate interruption
- 06Publish, then a reversible deploy
- 07The reported case, retested
Every step except the report is already automated, or already could be.
Closing the loop is a governance decision, not an engineering leap.
Remove that one step for a narrow enough class of fault and the circle closes. Somebody reports a problem in the afternoon; the problem is gone the next morning; nobody stood in the middle of it. This is not self-improving software in any grand sense. It is self-healing in a narrow and useful one: for a defined kind of fault, the cost of fixing it comes close to the cost of reporting it.
The question is not whether the loop can close. It is which faults you would let through it, and what you would want to see afterwards.
We do not run it that way, and most teams should not either. But the honest reason is not that something is missing from the machinery. Publishing without approval is already a setting. What is missing is the set of safeguards that would make using it a considered decision rather than an optimistic one.
- A kind of fault defined in advance rather than judged case by case: a failing test demonstrates it, the fix stays inside the permitted area, the change stays small.
- Tests the team would genuinely stake a release on, and an automated build that has to pass before anything ships.
- A release you can undo, and an automatic reversal if the fix turns out to break something else.
- A limit on how much goes through: one at a time, a few a day, never during a code freeze.
- Notice and ownership. Whoever owns that part of the product learns it changed, and stays accountable although they pressed nothing.
- The reporter's own case as the measure of success. The loop has closed when their problem is gone, not when the tests are green.
- A record of what changed and why, kept well enough to reconstruct the decision months later.
Most of that list is work, and several items sit in the section below rather than in the product today. The point is not that it is solved. It is that the shape is already there: the sequence runs from one end to the other and stops in one deliberate place. Closing the loop is therefore a decision about which changes you trust, not an engineering leap.
Which also suggests the right way to move. Not by switching approval off, but by shrinking the set of changes that need it, one defined kind at a time, and watching what comes back round.
What it does not do
- Only the last step depends on where you keep your code. Recording a change is ordinary version control — the same Git nearly every development team already uses — so any provider serves. Submitting it for review goes through that provider's own interface, and Bitbucket's is the one built today.
- Bringing work in from a tracking tool is likewise one integration rather than a general one, though it asks very little of the tool: somewhere to mark an issue as a bug is most of it.
- Changes are always offered as drafts, and the system never accepts its own work. Assigning reviewers, routing by ownership and sending notifications are not built yet.
- It waits for the automated build by checking repeatedly, which ties up one of its own working slots. Having the build report back instead is the next piece of work.
- Screenshots for visual fixes need a preview of the change and a browser available to look at it. Without them, the proposed change is the only evidence a reviewer gets.
- The control room is for administrators. There are no per-system reviewer roles and no delegated approvals.
- Unclear faults, design decisions, and anything that needs a conversation with the person who reported it are assessed and left for a human.
We state that list deliberately. A service like this is easy to oversell, and the fastest way to lose a team's confidence in one is a claim it cannot keep on a Tuesday morning.
Where it fits, and how to start
The pattern earns its place where three things are true at once: the code cannot leave the organisation's own network, the computing capacity is already owned, and there is a queue of small faults that will otherwise never be scheduled. Regulated organisations, public-sector platforms and product teams running their own AI infrastructure tend to have all three.
1. Watch it think, and nothing more
Let it assess incoming reports with no access to the code at all. Compare its priority, its summary and its guess at the affected area against what the team concluded on its own. This costs almost nothing, and it calibrates how much the later stages deserve to be trusted.
2. One system, one narrow area
Let it prepare fixes, with approval required, a small permitted area, a real set of tests and the lowest-priority queue. The useful measure is not how many changes were accepted. It is how many proposals a reviewer opened, and how many of those they trusted enough to take forward.
3. Widen the boundary, not the autonomy
As the evidence accumulates, widen the permitted area, raise the size limit and add systems. Move the approval point last, and when you do, move it for one defined kind of fault rather than in general — with the safeguards above in place, and the reporter's own case as the test of whether the loop really closed. Most of the value arrives long before that: a person still decides, and the decision now takes ten minutes instead of an afternoon.
The result is quieter than it sounds. Nobody arrives to software that changed overnight. They arrive to a short queue of prepared work with the evidence attached, produced by capacity the organisation had already bought and was not using. The distance between someone noticing a problem and someone being able to fix it gets shorter, and nothing about who decides has changed.
- Does this replace developer review?
- No. Every change arrives as a draft and goes through the team's normal review before it is accepted. The service prepares work and attaches the evidence; it does not approve or accept anything.
- Why run it overnight rather than during the day?
- Two reasons. On capacity you already own the hours are otherwise idle, so a long attempt with several retries costs nothing it would not have cost anyway. And work that is ready before anyone arrives costs the team no interruption, which is what made the fault unattractive to schedule in the first place.
- What stops it from changing something sensitive?
- It can only read and change the areas it has been given, its change has to stay under an agreed size, and protected files — deployment settings, build configuration, credentials, anything sensitive — are refused outright. Then the team's own tests run in a sealed copy, optionally the team's automated build, and a person approves before anything reaches the real code. An emergency stop halts anything leaving the system immediately.
- Could it fix and release without anyone involved?
- Mechanically, yes — every step except the report is already automated or could be, and publishing without approval is a setting rather than a missing feature. Whether you should is a different question. It needs a kind of fault defined in advance, tests you would stake a release on, a release you can undo automatically, a limit on how much goes through, notice to whoever owns that part of the product, and the reporter's own case as the measure of success. Until those hold, the approval step is doing real work.
- What kind of faults does it actually fix?
- Small, repeatable, well-located ones: a boundary handled wrongly, a wrong default, an empty case nobody covered, a misleading validation message. Faults that need a design decision, or a conversation with the person who reported them, are assessed and left for a person.
Newsletter
Occasional notes on building systems that hold up
A short email when we publish something worth your time. Architecture, integration, and operational AI in regulated organizations. No cadence promises, no forwarding your address.