Questions, answered short.
One page for the diligence reviewer. Each answer is a summary; the link beneath it goes to the page that holds the detail.
Jump to: Technical evaluator, Procurement, Security.
How it works and how you control it.
Who is running this today?
Colony runs Colony’s own development, with public receipts on Built by Colony. It is in production with an enterprise customer. Details of that engagement are under NDA; the public ledger shows the same pipeline’s behavior.
Is this just a wrapper around GitHub Copilot, Claude Code or Cursor?
Colony is the orchestration layer. It picks issues out of a backlog, runs analysis and planning before code is written, executes the implementation, reviews the result against your conventions, and merges it through your branch protection. The model that drives the developer agent is swappable. The orchestration is the product; the model is a dependency.
Does this replace developers?
No. Colony expands your team; it does not replace it. Teams that deploy it successfully redirect developer capacity toward work that needs human judgment, such as architecture, technical strategy and exploratory work. Teams that use it to justify headcount reduction typically see quality degrade as the human review layer thins.
What happens to our developers?
Their day-to-day shifts. Engineers keep their IDE workflow for interactive work and choose what goes into the pipeline: bugs they noticed, refactors they sketched, scoped features. Implementation moves to the pipeline; their time moves toward design, architecture and ambiguity, plus reviewing what the pipeline produces.
What stops the agents from doing something we don’t want?
Three configurable controls: per-issue cost caps that escalate to a human, an auto-merge switch that is off by default, per repository, and human-review-required labels that route matching issues through mandatory review. The Inspector enforces evidence requirements, and the Marshal respects branch protection. Start conservative and loosen as the escape rate proves out.
Where does our source code actually run?
Three options. Colony Cloud runs managed worker pools in our infrastructure, single-tenant per customer. Hybrid tenancy — control plane managed by us, worker pools in your VPC — is in private preview, available by arrangement. Self-hosted means you run the whole thing yourself. In every case, we don’t train on your code.
What does week one of a pilot look like?
Pick one repository with a single owning team and a well-defined backlog, and designate an Operator to own the conventions file. Start Colony with conservative risk envelopes, hold a 30-minute team orientation, and establish metric visibility from day one. Most pilots are issuing their first PR within a week.
Our tickets have always been loose specs. Is that a blocker?
Not a blocker. The Surveyor agent requests clarification when an issue is underspecified. The fastest fix is LLM-assisted issue drafting, which turns plain-language intent into the five-element format the pipeline needs. Teams that adopt this workflow get clarification rates near zero within two weeks.
How do we measure whether it’s working?
Six metrics, read together: throughput, cycle time, cost per issue, escape rate, agent attribution and review cycles. The most common mistake is optimizing for one metric in isolation. Throughput up and escape rate up means broken code shipping faster.
Spend, invoices and data.
How do you attribute LLM token spend back to a specific issue?
Every agent invocation carries the issue ID through the pipeline. Each prompt and completion is logged against that ID with the provider, model, input and output tokens, and the unit price at the time of call. Rollups by issue, agent, repo, tenant and month are precomputed views over the underlying log.
Can finance get an export they can reconcile against the model-provider invoice?
Yes. The Ledger export is a CSV with one row per agent invocation, including tokens and unit price. Sums by provider and month line up with the provider invoice within rounding.
Do you mark up tokens? Where is the markup visible?
Token cost is pass-through at the provider unit price plus a small markup, set on the contract and visible as a separate column in the Ledger. Worker capacity, platform tier and managed services are priced separately.
How long is data retained, and can we delete it?
The Ledger is retained for the period required by jurisdiction for finance and tax purposes. Prompts and completions can be deleted on request within thirty days with a signed deletion certificate, separate from the cost rollups required for audit.
Can we run our own dashboards against the data?
Yes. The Ledger is queryable Postgres views with a stable schema. Wire it into your BI tool, alert on spend per agent, and build a dashboard your CFO actually reads.
What an auditor would ask.
Do you train on our code, and which model providers do you use?
Colony talks to model providers (Anthropic, OpenAI) through their enterprise APIs, using the endpoints that contractually exclude prompt and completion data from training. If you have a procurement preference, we can pin agents to a provider on Cloud, or you can configure your own keys on the self-hosted deployment.
Where is our data stored, and can it stay in our VPC?
Colony Cloud runs in US-East and EU-West today, and tenants choose at provisioning time. We do not move tenant data across regions. Hybrid tenancy — control plane managed by us, worker pools in your VPC — is in private preview, available by arrangement. It suits teams with stricter residency requirements; the control plane never touches code.
What happens to our code while the agents work on it?
The Builder agent checks code out into ephemeral workers, which are torn down after the issue completes. We do not persist working copies beyond the issue’s lifetime. Only the diff, the prompts and the responses are retained in the audit trail, scoped per tenant and encrypted at rest.
What GitHub permissions does Colony need, and can it bypass branch protection?
Colony installs as a GitHub App with read on issues and code, and write on pull requests, branches and statuses. We never request organization-wide admin scope. The Marshal agent uses the same merge surface a human would, so required reviews, status checks, code-owner approvals and signed-commit requirements are all honored.
What audit record do we get?
A signed, timestamped audit record for every issue: who authored it, which agent did which work, when each transition happened, which human reviewer approved the PR, and the full cost attribution. It is available as a CSV export. For compliance-sensitive paths, leave auto-merge off on that repository (the default), so every PR requires explicit human approval.
Are you SOC 2 certified?
Not yet. Colony Cloud has not yet completed a SOC 2 Type II audit. The audit is on the roadmap; we’ll publish the timeline once it’s scheduled. What we publish describes the controls themselves rather than a third-party attestation. Penetration testing is on the roadmap; we’ll name the firm and the date once the first engagement is scheduled.
What happens if we end the engagement?
You can revoke the GitHub App at any time, and the pipeline immediately stops. On request, we will delete all retained prompts, completions and audit data within thirty days, with a deletion certificate signed by an officer. The cost ledger is retained separately for finance and tax purposes.
A question not on the list?
Bring it to the pilot call. We answer specifically; we don’t hand-wave.