“Postgres is the authority for pipeline state; GitHub labels are a projection for human visibility.” That line is in Colony’s architecture documentation now. It earned its place through a specific failure pattern that label operations alone could not prevent.
Two consistency models
Postgres and the GitHub API have fundamentally different consistency properties — a design constraint that determines which one can be trusted as a state machine authority.
A Postgres UPDATE inside a serializable transaction either commits or it doesn’t. A concurrent reader using SELECT FOR UPDATE sees a consistent view and blocks until the writer releases the lock. Invariants — that an issue’s blocked flag matches its state, that only one worker claims a given task — can be enforced at the database level without coordination between callers.
The GitHub API is eventually consistent. Octokit’s HTTP client caches responses using ETags. When a process calls GET /repos/{owner}/{repo}/issues, the server can return 304 Not Modified based on the cached ETag — and the ETag computation for paginated issue-list responses does not always reflect label changes immediately. A label applied one minute ago may not appear in the next polling response.
For a web app where a human is reading the issue tracker, that gap is irrelevant. For a pipeline routing decisions on label state across multiple concurrent workers, it compounds quickly.
The race
Colony runs several classes of processes: the Mayor scans for state changes and enqueues tasks; worker containers execute analysis, development, review, and merging. Both classes read issue state to make routing decisions, and in the early architecture, both read that state from GitHub labels.
The failure pattern developed as the worker pool grew. A worker would apply the colony:blocked label to an issue after detecting a dependency conflict. The Mayor — running in a separate container with its own Octokit client and its own ETag cache — would poll the issue list on its next cycle. The conditional GET returned 304 Not Modified, because the server’s ETag for that list hadn’t invalidated yet. From the Mayor’s perspective, the blocked issue still looked active. It re-enqueued a task. A worker claimed the task, read the issue state, found the block, and did nothing useful. The Mayor polled again and re-enqueued again. Comment floods accumulated. Workers burned through budget on no-ops.
Two earlier incidents made the problem worse. First, the labels array in the pipeline_issues Postgres table was not updated during every state transition — it was a separate write that could fall out of sync with the actual GitHub labels. The Mayor read the DB labels column for some routing decisions and queried GitHub labels for others. Both sources were stale in different ways. Second, when an issue was unblocked and resumed pipeline work, the colony:blocked flag label was not always stripped. Issues progressed through development and review carrying a stale blocked marker, which caused routing logic to behave inconsistently depending on which label read it hit first.
Three separate incidents, same root cause: the pipeline had two sources of truth for issue state, and they diverged under concurrent writes.
Moving to Postgres-first
The architectural change was to make Postgres the single authority and demote GitHub labels to a read-only projection.
The Mayor now reads pipeline_issues.state, is_blocked, and is_paused directly from Postgres for all routing decisions. It uses isBlockedInPg and isPausedInPg accessors against the Postgres row. GitHub label reads are limited to one narrow role: detecting human-initiated changes, where an operator manually edits labels on the GitHub issue tracker, and reconciling those changes back into Postgres.
All state transitions go through transitionIssuePgFirst(), which writes Postgres first, then projects labels to GitHub as a side effect via syncLabelsFromPostgres(). Labels are computed from the current Postgres state and applied to GitHub after the Postgres write succeeds. Labels are an output.
The function that enforces the invariants is atomicTransitionWithSnapshot(). PipelineStore exposes it as a thin passthrough method in pipeline-store.ts; the implementation lives in transition-store.ts. Rather than checking a static allowed-transition table, it validates the requested transition against a WorkflowDefinition snapshot passed in by the caller — and that check runs first, before the transaction opens: an illegal transition is rejected without a write. Within a single Postgres transaction:
1. SELECT FOR UPDATE the issue row; verify the current state matches, the pinned workflow matches the supplied snapshot, and the issue isn't paused
2. UPDATE state, state_entered_at, is_blocked (is_paused is managed separately, by operator action)
3. Cancel pending and claimed work_tasks when the target state is blocked or terminal
4. INSERT a new work_task if the target state has an associated executor
5. Queue GitHub label projections for the new state
6. INSERT state_transitions audit recordThe invariants that were previously maintained by callers are now enforced by the transaction: is_blocked is set to true if and only if the target state’s type trait is blocked — which covers DependencyBlocked and any state in a custom WorkflowDefinition carrying that trait. Pausing runs through its own dedicated transaction in a separate lifecycle store, so its invariant is enforced there rather than inside this one. Before this change, there were more than six separate callers updating the blocked flag and more than five separate callers running deletePendingTasksForIssue(). Each made independent decisions about when to run those operations. Any one could fail, be skipped, or run in the wrong order without the others knowing.
After: one function, one transaction. If the state update commits, the flags, audit record, and task queue update commit with it. If any step fails, the whole transaction rolls back. There are no partial states.
One exception remains: when an operator applies the colony:blocked label directly on GitHub, the webhook receiver reconciles that human-initiated change into Postgres via a standalone setBlockedFlag() write that sits outside the state machine. That path is documented explicitly as outside the atomic pattern — it’s the same narrow “detect human-initiated changes” role GitHub labels were demoted to. The general rule holds everywhere else, including the auto-merge retry-limit path, which now runs through the same atomic transition as every other routing decision.
What this enables
Cost cap atomicity. Per-issue budget enforcement can now happen in the same transaction as the state transition that dequeues a task. The window that previously existed between “check passes” and “task is claimed” — where a second worker could also pass the check and both proceed — closes when both operations are inside the same transaction.
Audit trail integrity. Every state transition inserts a row into state_transitions with from-state, to-state, agent name, and reason. Because this insert is inside the atomic transition rather than a separate call that might fail independently, the table is a complete and consistent record. Before the migration, Colony’s dashboard Activity Feed was blank: it sourced data only from in-memory agent health snapshots that reset on agent restart, leaving the state_transitions table unread. After the migration, the feed reads from the authoritative state_transitions table and shows a complete timeline.
Replay and cycle time analysis. The state_transitions table is an append-only event log. Reconstructing the full lifecycle of any issue — every agent that touched it, every state it passed through, when it was blocked and when it was unblocked — is a SQL query against this table. This is the data source behind per-issue cost attribution and the failure pattern analysis in Colony’s production learnings.
Elimination of the race class. The ETag staleness, cache divergence, and propagation delay failures all had the same structure: two or more processes reading a mutable external store and acting on their individual snapshots. Centralizing authoritative state in Postgres doesn’t eliminate every race condition in the pipeline — there are other operations that interact with GitHub — but it eliminates this specific class entirely. Every routing decision the Mayor makes now reads from a store that provides isolation guarantees. Previously it read from an HTTP response that may be stale by design.
The broader principle
The GitHub API is well-suited to its intended purpose: human-readable issue tracking, webhook-triggered automation, manual label management by engineers with a browser open. A state machine under concurrent write load from multiple containerized workers requires stronger consistency guarantees.
Colony’s pipeline now uses each system for what it is actually good at. Postgres enforces invariants, serializes transitions, and provides a consistent read for every routing decision. GitHub labels reflect current Postgres state for operators looking at the issue tracker. The humans see labels. The pipeline reads a database.
The question “what is the current state of this issue” has one answer now, and it comes from one place.
If your pipeline routes decisions on state that more than one process can write — and the answer to “what is the current state” comes from somewhere eventually consistent — we’d like to talk. We should talk about yours. The pipeline is at the self-hosted pipeline.
If you’d like to see the pipeline running on your work, we should talk.