We found defects in 37 of DeepSWE’s 113 tasks

A human-reviewed audit found defects or ambiguous requirements in 37 of DeepSWE v1.1’s 113 tasks (32.7%), with examples of how the evaluations went wrong.