Interactive testing improves coding agents' application repairs in preprint

Researchers report that coding agents repaired 75.5% of broken application requirements within five attempts when feedback came from agents interacting with the software, compared with 55.4% using code-only feedback. The preprint tests an evaluation method in which computer-use agents operate websites, desktop applications and games to assess their behavior. The researchers released 6,858 application pairs with human preferences and agent traces, plus 222 applications with human-verified requirement labels. Interactive evaluation improved prediction of human preferences from 65.6% to 69.1% and requirement verification from 77.0% to 81.6%. The method improved alignment with human judgments, but its preference predictions still fell short of matching them consistently.