The usual advice for coding agents is to write the test first and let the agent work until it passes. We ran that set-up thirteen times on real work, with a second agent reviewing each result, and kept a table of the defects. The test written first caught almost none of them, and was often wrong itself. Three things follow for anyone who works this way:
Check the test before the work starts. The agent that wrote the task and its test was itself a large source of errors, and the test was wrong in about half the runs, usually by passing while the thing it guarded was broken. A test for a fix has to hold both sides. In one run a fix passed its tests by stepping aside from the question: the wrong answers were gone, and so were many of the right ones. Give the reviewer the result, not the worker’s account of it. A reviewer who was not given the worker’s report found most of what was found, including false sentences in results whose numbers were all correct. The numbers and their limits follow. The agents are Anthropic’s Claude models, not local ones. The counts, the table behind them and a script that recomputes them are public.
...