Methodology

Tasks

Every task comes from SWE-bench Verified (SWE-bench/SWE-bench_Verified, revision 78f471bf65), a set of 500 real GitHub issues from popular Python projects that human annotators confirmed are well-specified and fairly tested. We use the 194 tasks the annotators rated “<15 min fix”, so the 15-minute limit is plausible.

Each candidate was then validated on a GitHub-hosted runner with the same harness used for scoring: the reference fix must resolve it, and a no-op patch must not (otherwise the tests can't tell a fix from no fix). 192 passed. From those, 100 were drawn, stratified by repository so the mix matches the pool:

Left out: django__django-10097 (failed validation: gold=resolved, no-op=resolved); psf__requests-2317 (Its tests make live network calls; the evaluation hung on GitHub-hosted runners instead of finishing.).

Each person gets the tasks in their own random order (seeded per account), one attempt per task, with no skipping. That stops cherry-picking easy tasks and spreads human coverage across the whole set.

Environment

You work in a browser editor over the full repository at the task's base commit: file tree, quick open, full-text and regex search, multiple tabs, new files. Binary files, translations and very large files are listed but not loaded. You see the issue text exactly as the models did. Hints from the issue thread are hidden (the leaderboard doesn't give them to models either) and revealed afterwards.

The clock is 15 minutes, measured on the server from the moment you press Start. The repository downloads before the clock starts. At 0:00, whatever you've changed is submitted automatically; if you close the tab, your last autosaved draft (saved every 10 seconds) is submitted for you.

You cannot run code. That's the biggest difference from the agents, which could run tests and scripts in a shell. Running untrusted code for every visitor isn't something we can do safely for free. Keep it in mind when reading the numbers.

Scoring

Your edits become a unified git diff in the browser, checked by re-applying it to the original files before it's sent. Each submission becomes a pull request in the public defdevio/areyoutbetterthan.ai-submissions repository, where a GitHub Actions workflow runs the official SWE-bench harness (swebench==5.0.2) with the prebuilt task images, the same way leaderboard submissions are evaluated. Your diff is treated as data and only ever applied inside the task's Docker container. The workflow has no secrets.

• Resolved: every FAIL_TO_PASS test (the ones that reproduce the bug) and every PASS_TO_PASS test (regression checks) passes.
• Unresolved: anything else, including a patch that doesn't apply, tests that time out, and an empty submission.
• Infrastructure errors (Docker, runner, network) are never counted against you. They're retried automatically up to twice, then flagged for a human to fix.

Every PR and workflow log stays public as an audit trail. PRs are closed after scoring and never merged.

Model comparison

Per-task model results come from SWE-bench/experiments (commit 40f164d, fetched 2026-10-17), refreshed weekly. Of 173 submissions with per-instance results, the default view keeps one row per model (75): its most recent result, preferring the bash-only mini-SWE-agent scaffold, where every model gets the same minimal tools. That's the closest thing to an apples-to-apples model comparison. “All submissions” shows every agent+model entry instead.

Comparisons are always on the subset of tasks you attempted: if you did 12 tasks, each model is scored on those 12, not on all 500. A model that didn't produce a patch for a task counts as unresolved, as on the leaderboard. Entries whose per-instance data disagrees with their published score by more than 2 points are excluded.

A second, independent source is vals.ai, which runs 86 models itself with the same bash-only harness and also reports wall-clock time and API cost per task (snapshot from 2026-09-01). Pick “vals.ai runs” on your results page to compare against it. Their time and cost figures are averages over all “<15 min fix” tasks rather than per task, so they describe the same difficulty band, not exactly your tasks.

Metrics

The headline is the resolve rate, since that's what the leaderboard ranks on. We also record:

• Time: from pressing Start to submitting, measured on the server. Auto-submits and abandoned attempts count as the full 15 minutes. We show the median per task and the median for correct fixes only, and a “solved within N minutes” curve.
• Bug tests passing (partial credit): the share of each task's FAIL_TO_PASS tests your patch makes pass, averaged over attempts. No patch or a patch that doesn't apply counts as 0.
• Found the right file: whether your patch changed at least one file the upstream fix changed, plus file-level F1 against it. It's a measure of locating the bug, independent of fixing it.
• Patch size: lines you changed relative to the upstream fix.
• How it ended: resolved; fixed the bug but broke other tests; partly fixed; didn't fix it; patch didn't apply; no patch; or tests timed out.

For models, only resolve rate (both sources) and vals.ai's average time and cost are available. Human and model times aren't perfectly comparable: humans are capped at 15 minutes and time includes reading the issue; model time is uncapped and includes API latency.

Statistics

Rates come with Wilson 95% intervals, which behave sensibly for small samples and rates near 0% or 100%. With a handful of tasks the intervals are wide, and we say so. “You beat N models” compares point estimates; overlapping intervals mean the difference may be noise. The global page pools all non-flagged attempts and also shows the mean of per-task human rates, so prolific players don't dominate. It stays hidden until there are enough attempts to be meaningful.

Honor system

Please don't use AI assistants or look up the actual fix. The reference patches are public, so we can't make cheating impossible. We don't block anything, but we record a few signals and flag attempts that trip them:

• a single paste of 400+ characters from outside the workspace,
• 1000+ characters pasted from outside in total,
• a patch of 4+ changed lines submitted within 45 seconds.

Copying and pasting within the editor or from the issue doesn't count. Flagged attempts still appear in your own results but are excluded from the global human statistics.

Limitations

• No code execution for humans; agents could run code.
• A 15-minute cap for humans; agents had no wall-clock limit.
• The people who show up are self-selected and probably above-average programmers.
• Some participants may already know a task's fix; the issues are public.
• Only the easiest SWE-bench Verified difficulty bucket is used.
• Model results were produced by their submitters, sometimes with older harness versions. We take them as published.

Results are reported as they are, whichever side they make look good.

Data & privacy

We store your GitHub id, login and avatar URL; your attempts (start/end times, your diff, paste sizes and timings, but not paste contents); and scoring results. The GitHub sign-in token is used once to read your public profile and is never stored. Your diffs are public in the submissions repository, attributed to your GitHub login. Share pages are opt-in and show only the display name you pick.

Task snapshot selected 2026-10-17. Dataset fetched 2026-10-17.