Evals — Astroloop Pro

Results

Pro only

Measured against a person doing the same 412 issues.

Including the row where we lose. Methodology is below the table, not behind a form.

Task	Astroloop	Baseline	n	Note
Issue to merged pull request	68%	41%	412	Human review still required; measures reaching an approvable PR.
Correct on the first attempt	44%	39%	412	Retries are free, so this is a cost metric rather than a quality one.
Escalated to a person	31%	12%	412	Deliberately high. Every escalation is a run we chose not to guess on.
Actions taken without approval	0	7	412	Enforced in the tool layer, so this is a property rather than a result.
Median wall clock per run	214s	96s	412	We are slower. Planning and re-reading cost time we choose to spend.

Methodology

Where did the 412 issues come from?: Four open-source repositories with more than 200 stars each, sampled from issues opened in the six months before the run and closed by a human at the time of sampling.
What is the baseline?: The original human-authored fix, measured from the first comment on the issue to the merge commit. It is a generous baseline: those people had context we did not give the agent.
Who judged correctness?: Two reviewers independently, blind to which fix was which, with a third breaking ties. Agreement before tie-breaking was 91%.
What counts as escalated?: Any run that stopped and asked rather than producing a pull request. We count these against ourselves deliberately — a system tuned to escalate less is not a better system.
What is not measured here?: Anything requiring product judgement, anything touching more than four files, and anything where the issue itself was ambiguous. Those were excluded before the run, not after.
Can I reproduce it?: The issue list, the prompts and the raw transcripts are published alongside this page. The one thing we cannot give you is the reviewers.

Twenty runs a month, free, with the same permissions and the same transcripts as every other plan.