We found defects in 37 of DeepSWE’s 113 tasks

DeepSWE v1.1 is used to compare some of the most capable AI models available. It appears in OpenAI’s GPT-6 Astra launch table and Anthropic’s Fable 5.1 system card. OpenAI’s table reports scores of 74.1% for Astra, 73.7% for Opus 5, 72.7% for Sol and 69.9% for Fable 5. Differences of a percentage point or less appear to separate several of these models.

Yet when we reviewed all 372 recorded failures for Opus 5, Sol and Fable 5, we found defects or ambiguous requirements in 37 of DeepSWE’s 113 tasks (32.7%). These included hidden tests that broke the build, assertions that rejected valid output, and tests that enforced requirements the instructions left unclear.1

One task asks the model to add typed variables to a scripting language implemented in Go. The prompt encourages testing, and the agent added a test for the new feature.

After the agent submitted its solution, the benchmark injected hidden tests into the same Go package to evaluate it. Those tests were unavailable to the agent while it worked. The prompt did not describe this injection step or reserve any test names:

// Agent's test, written before submission
func TestTypedBindingsDeclarations(t *testing.T) { ... }

// Hidden test, injected during evaluation
func TestTypedBindingsDeclarations(t *testing.T) { ... }

Declaring two functions with the same name in the same package is a compilation error, so adding the hidden tests broke the build. The agent had no opportunity to see or resolve the collision; the benchmark recorded a failure without testing the feature.

Renaming the hidden tests allowed all three affected submissions to pass. This was part of a wider pattern: Go naming collisions caused 18 false failures across five tasks.

Other failures came down to a single word. One task required invalid configuration values to produce a warning through npmlog, with the prefix bail_on_test_failure, and then fall back to false. The submission did this:

log.warn('bail_on_test_failure',
  'Expected true, false, or a positive integer; defaulting to false.');
return false;

But the test also required the warning to contain “invalid”:

expect(warnStub.firstCall.args[1]).to.match(/invalid/i);

The prompt specified the logging library, prefix and fallback, but did not prescribe the message’s wording. The warning explained the accepted values and the fallback. Four Sol submissions failed solely because “invalid” was missing. We classed this as a defect: keeping the warning and fallback checks while removing the word requirement made all four pass.

A SQL formatting task shows the same gap in a different form. Eight submissions failed tests that required IN ( to become IN(, but the prompt did not settle whether that space should remain. We classed this as an ambiguity rather than a defect: the test is not wrong, but a formatting test needs an expected style, and the instructions must tell the agent what that style is.

Other examples:

Tallied by task:

Fixing the confirmed defects and rerunning the original submissions turned 67 failures into passes without changing the submitted code. This raised the three models’ measured pass rates by 4.22 to 6.19 percentage points:2

† Hypothetical: adds all 98 ambiguous failures to the verified passes. The requirements must be clarified and the submissions rerun to test this assumption. pp = percentage points.

The larger problem is that these evaluation problems remain in a benchmark cited as evidence of frontier capability. A score cannot distinguish an implementation error from a scoring error or an unstated requirement. Optimize against such scores without reading the failures, and you train models to satisfy the evaluator rather than solve the problem.

As coding benchmarks approach saturation, the remaining gap to 100% becomes harder to interpret. Before counting another percentage point as progress, we should check what the point was awarded for.

1. Codex assisted with the review of all 372 recorded failures; a human expert reviewed the 180 flagged cases. We tested evaluator corrections on the archived submissions without generating new solutions, using published task images.

2. The archived runs number 444 for Opus 5, 450 for Sol and 436 for Fable 5. Corrections preserve those populations and count only unchanged submissions that passed all required tests after repair. The final column assumes all 98 ambiguous failures also become passes; this has not been tested. Fable’s archived baseline is 69.72%, rather than the launch table’s 69.9%. We did not audit Astra or Fable 5.1.