Humans vs AI
Every human attempt, pooled, against every model, on the same real GitHub issues graded by the same tests.
Every human attempt, pooled, against every model, on the same real GitHub issues graded by the same tests.
| # | Solver | ||||
|---|---|---|---|---|---|
| ? | Humansranked once 10 attempts are scored · 1/10 so far · add yours → | ||||
| =1 | Claude Opus 5Anthropic | 99.0% | 98.5% ±0.9 | 6m 51s | $0.88 |
| =1 | Claude Fable 5Anthropic | 99.0% | 95.9% ±1.4 | 4m 21s | $0.41 |
| 3 | GLM 5.3Zhipu AI | 98.0% | 97.4% ±1.1 | 10m 29s | $0.23 |
| =4 | DeepSeek V4 Pro 0813DeepSeek | 97.0% | 97.4% ±1.1 | 3m 05s | $0.076 |
| =4 | GPT 5.6 SolOpenAI | 97.0% | 97.4% ±1.1 | 2m 23s | $0.93 |
| =4 | GPT 5.6 LunaOpenAI | 97.0% | 96.4% ±1.3 | 2m 51s | $0.032 |
| =4 | Grok 4.6SpaceXAI | 97.0% | 95.9% ±1.4 | 7m 56s | $0.57 |
| =4 | Muse Spark 1 2Meta | 97.0% | 92.3% ±1.9 | 5m 51s | $0.40 |
| =9 | GPT 5.6 TerraOpenAI | 96.0% | 96.9% ±1.2 | 2m 33s | $0.32 |
| =9 | Grok 4.5SpaceXAI | 96.0% | 93.8% ±1.7 | 2m 11s | $0.33 |
| =9 | Inkling SmallThinkingmachines | 96.0% | 93.8% ±1.7 | 2m 48s | $0.088 |
| 12 | Claude Opus 4 8 Claude CodeAnthropic | 95.0% | 91.2% ±2.0 | 1m 11s | $0.38 |
| 13 | GPT 5.5OpenAI | 94.0% | 91.8% ±2.0 | 5m 39s | $0.59 |
| =14 | Qwen3.8 MaxAlibaba | 93.0% | 93.3% ±1.8 | 30m 37s | $0.60 |
| =14 | Claude Opus 4 8Anthropic | 93.0% | 92.8% ±1.9 | 6m 58s | $0.72 |
Head-to-head scores each model on exactly the attempts humans made (a task three people tried counts three times), so it's directly comparable to the pooled human rate. All 100 tasks is each model on the whole task set. = marks a tie on the sorted column; tied rows are ordered by published accuracy. Published accuracy, time and cost are vals.ai's figures for the “<15 min fix” band our tasks come from. Human time is capped at 15 minutes and includes reading the issue; model time includes API latency.
Time, partial credit, and where humans find the bug appear here once 10 attempts are scored (1 so far).
| Task | Humans | vals.ai models | Leaderboard models |
|---|---|---|---|
QuerySet.ordered property is incorrect for GROUP BY queries on models with Meta.ordering. django__django-13417 | 0/1 | 82/86 | 59/75 |
Flagged attempts (honor-system checks) are excluded. Updated every few minutes. How models and humans are compared →