Alibaba Benchmark Finds AI Coding Agents Need Human Rework on Most Tasks
A new, more realistic benchmark from Alibaba tests AI coding agents with real user requests. The top-ranked model, GPT-6 Astra, fully solves only 26.4 percent of the tasks.
Despite strong results in established tests, AI coding agents still require human rework in practice: on roughly three out of four tasks, developers have to intervene. This is shown by a new benchmark called Co-Design Bench, which Alibaba has published, as The Decoder reports.
The test is based on real user requests that had previously led to errors in AI agents. In this way, it takes a different approach from many earlier evaluation methods, which work with curated and cleanly tailored scenarios. The aim is to depict realistic rather than idealized programming situations, that is, precisely the tasks developers face in their everyday work.
The result is markedly more sobering than in older benchmarks. The leading model, GPT-6 Astra, fully solves 26.4 percent of the tasks. According to the available information, the remaining roughly three out of four tasks require human rework. In older test series, the same models, by contrast, achieve scores of over 99 percent. This discrepancy suggests that older benchmarks may overestimate the actual programming ability of such systems in everyday use.
It remains open how many tasks the test includes and how their difficulty is distributed. It is also unclear by which criteria a task is considered fully solved, which makes the figure of 26.4 percent difficult to interpret on its own. No further details are available on the performance of models below the top position, so the comparison in the mid-field can only be judged to a limited extent.
For companies that already integrate AI agents into software development or plan to do so, the finding is relevant. It indicates that autonomous programming aids can prepare and speed up work, but a reliable handover without human oversight currently rarely succeeds. Realistic evaluation methods such as this one are likely to become an important basis for assessing such tools, because they show where the capabilities of these systems actually end.
The comparison between the benchmarks also points to a fundamental measurement problem: when a model works almost flawlessly in one test and, in another that draws on real user requests, fully solves about a quarter of the tasks, the result of the first says little about its performance in the second. The gap between the two measurements is therefore a sign that benchmarks will need to be aligned more closely with the working reality of developers in the future if their results are to serve as a basis for decisions.
Source: the-decoder.com
2
