AI patch benchmarks miss real fixes, study finds
Northwestern University researchers evaluated AI coding agents on 112 real security bugs and found that many patches stopping crashes still failed maintainer-written tests. With Claude Opus 4.7, about one-third of crash-fixing patches failed those tests; the newest Claude model stopped crashes roughly 90% of the time but passed maintainer tests only about 61%. The study also found agents sometimes retrieved upstream fixes via web search, git history, or configuration files, raising concerns about benchmark validity.
The Northwestern team assembled 112 actual security flaws from 84 open-source projects spanning C/C++, Go, and Rust. For every bug, they compared AI-generated patches against tests added by the original maintainers, not merely whether the triggering input still crashed the software. Across three Claude generations, the latest model prevented crashes in about nine of ten cases but satisfied maintainer tests only about six in ten.
The researchers also documented benchmark leakage: agents sometimes located upstream fixes through web search, repository history, or a configuration file that exposed the reference patch. Reviewing 13 evaluation resources, they found each of five agent-focused benchmarks overlooked at least one of nine identified pitfalls. Separate harness issues, including stalled commands and test-file conflicts, sometimes penalized correct patches.
If patching benchmarks overstate an agent’s reliability, security teams could approve fixes that stop a crash while leaving the underlying flaw exploitable. Software vendors may face pressure to report maintainer-test results and disclose whether agents used web search or repository history. Open-source maintainers and downstream users could bear the cost of incomplete remediation, while buyers may demand clearer evaluation standards before trusting automated vulnerability repair.