A retry loop with no cycle limit runs until something outside it stops it. In an autonomous pipeline that something is usually a bill, or a human who notices the notifications.
Colony hit this three times, at three different layers, before we wrote the rule down. Each incident had its own cause. Each fix landed in a different package. The design rule they converged on is short: every looping failure path needs a circuit breaker, and every breaker needs a manual-unblock escape hatch.
Two loops, one shape
Issue #816 was the PR-open → conflict → close loop. A PR opened, hit a conflict and closed, and the cycle started again. Nothing counted the cycles, so nothing ended them. The production learnings record the in-development → blocked → retry loop as open to the same indefinite repetition.
Issue #1067 showed the same shape at the task layer. While we were implementing the dependency graph, one run produced 84 failed tasks in three minutes. Every task hit the identical worktree error, failed, and was picked up again. The retry logic treated each failure as a fresh event, so the eighty-fourth attempt looked exactly like the first.
Both are recorded in colony/docs/production-learnings.md under “Circuit breakers are required at every looping failure path.” The pattern they share: the system has an accurate record of every failure and uses none of it to decide whether to try again.
The build and review guards
The build-failure breaker shipped in #1089. It lives in developer/src/guards.ts, with the decision point in pre-flight.ts. It counts consecutive identical failures, and “identical” has a precise meaning in the code. Each guard declares its threshold as an exported constant:
export const MAX_CONSECUTIVE_IDENTICAL_BUILD_FAILURES = 3;
export const MAX_CONSECUTIVE_IDENTICAL_REVIEW_FAILURES = 2;The two constants sit in different parts of guards.ts, next to their own signature extractors. The build guard scans the issue’s comments from newest to oldest and extracts an error signature from each self-validation failure comment: the first meaningful line of the error detail, with hex addresses and numbers replaced by placeholders and whitespace collapsed. The first failure it finds sets the target signature. Each following failure with the same signature adds one to the count. A failure with a different signature stops the count, and so does a Builder comment that is not a failure.
That design makes the breaker patient with progress. A Builder that fails on a type error, then fixes it and fails on a lint error, has made progress, and the counter starts over. A Builder that hits the same type error three times in a row has not. The Inspector’s side of the loop uses the same scan with a lower threshold of 2, and its signature leads with the name of the failing check.
Each scan skips comments from other agents and from humans, so they leave the count untouched.
A guard that can be wrong
A signature match proves the failure repeated, and a retry can still fix it. One common case: a branch built against an old base fails on a TypeScript symbol that exists on main. Rebasing fixes it. Blocking the issue would turn a recoverable failure into a stuck one.
So before either guard trips, pre-flight.ts calls diagnoseTsBuildError. When the missing symbols exist on main, the diagnosis reports a stale base, the guard logs it and skips the bailout, and the normal flow rebases the branch.
When the guard does trip, it emits a circuit_breaker event with the signature and count, and posts a comment that names the signature and says human intervention is required. The comment is the escape hatch: it tells the operator what to look at and which commands unstick the issue.
Tripping has to stop the cycle
A breaker that logs and keeps going is only a metric. Issue #1091 added blocked-flag gating so that a tripped breaker leaves the issue in a state the pipeline stops picking up. In the current code the review-side guards call blockIssue, while the build-failure guard emits its circuit_breaker event and returns an early exit that the pipeline routes.
Reset semantics
When a counter resets matters as much as when it trips. A counter that never resets turns one bad week into a permanent block. A counter that resets too eagerly never trips.
The guards in guards.ts have two explicit reset boundaries, and each one is a check inside the comment scan:
if (comment.body.includes(CYCLE_RESET_MARKER)) {
break;
}
if (
comment.body.includes(BOT_SIGNATURES.sprintMaster) &&
comment.body.includes('Retrospective')
) {
break;
}CYCLE_RESET_MARKER is posted when an issue re-enters the pipeline (#819). The Mayor’s retrospective comment (posted by the sprint-master package) marks the point where the failure cause was reclassified (#876). Both are comments in the issue’s history, so the reset is visible to a human reading the thread.
The third reset, a human changing state by hand, has no marker of its own. It works indirectly. Moving the issue back through the retry path clears the stale blocked flag, and the next pass starts from a pipeline that accepts work for that issue again. We list it separately because a human action looks like it should reset the counters directly. The indirect route shows up the first time someone unblocks an issue and expects a clean count.
The adjacent case: workspace preparation
The worktree error in #1067 was a workspace-preparation failure. The May ledger listed that path as an open gap (#1892).
That gap is closed. A transient preparation failure now rethrows into the worker’s own failure path in worker/src/outcome-processor.ts. That path applies exponential backoff, from BASE_RETRY_DELAY_MS (2 minutes) up to MAX_RETRY_DELAY_MS (30 minutes), and the delay doubles with each attempt:
const retryDelay = Math.min(
BASE_RETRY_DELAY_MS * Math.pow(2, task.attemptCount),
MAX_RETRY_DELAY_MS,
);The first retry waits 2 minutes and the second waits 4. Later delays of 8 and 16 minutes, then a 30-minute cap, apply only when max_task_retries is raised above its default of 3. Each failure is recorded first, then the path counts the issue’s failed tasks from the last 60 minutes. When that count reaches max_task_retries, the issue is blocked instead of re-enqueued, so under the default the third failure is the last.
Run the incident numbers against the new path. Eighty-four failures in three minutes would now be three failures, with 2 and 4 minutes of backoff between them (about six minutes plus run time), followed by a blocked issue. The error stays the same, and the pipeline stops paying for it.
The design rule
Three layers found this independently:
- The PR lifecycle needed a cycle limit for the open → conflict → close loop (#816).
- The worker needed a retry budget and backoff for tasks that fail on infrastructure (#1067, and the workspace-preparation gap tracked in #1892).
- The Builder and Inspector guards needed signature matching so that an issue stops on repeated identical failures and keeps going while the failures change (#1089, #1091).
Each loop surfaced in production, after it had cost something. The rule we wrote down afterwards is the one we now apply when adding any path that can re-enter itself: every looping failure path gets a circuit breaker with a manual-unblock escape hatch. The breaker keeps a runaway loop bounded. The escape hatch keeps a tripped breaker from becoming a permanent verdict, because an operator who can see the signature, the count and the commands can decide that the next attempt deserves a chance.
The checklist for a new retry path is short: what counts as the same failure, how many are enough, what stops the cycle when the limit is reached, what resets the count, and how a human gets the issue moving again. If any of those has no answer, the loop is still open.
If you run an autonomous pipeline and want to compare which of your loops have a breaker and which have a hope, we’d like to talk. We should talk about yours. The pipeline is at the self-hosted pipeline.