ClawCodex Scores 80.9% on Terminal-Bench 2.1 on Claude Opus 5

Executive Summary

We ran ClawCodex — our open-source, MIT-licensed Python rebuild of Claude Code — against Terminal-Bench 2.1, the verified 89-task benchmark that measures how well an agent gets real, end-to-end work done inside a terminal. Driving claude-opus-5 headless at maximum reasoning effort, ClawCodex solved 72 of 89 tasks: 80.9% pass@1 on a single run.

On the public Terminal-Bench 2.1 leaderboard, whose figures are k=5 averages, that number would slot in around third — behind Claude Code on Fable 5 (83.8%) and Codex on GPT-5.5 (83.1%), statistically level with the 79–80% cluster below it, and ahead of Claude Code on Opus 4.8 (78.9%) and Sonnet 5 (74.6%). The headline isn't that a rebuild beat the original — the top entry is still Claude Code, on Fable 5. It's that a faithful, open rebuild lands right in the mix, ahead of the reference implementation on the same tier of model.

What Terminal-Bench 2.1 Measures

Terminal-Bench is a benchmark for agents that do things in a shell: compile a real project, recover a corrupt database, crack a hash, fix a git repository, build a Cython extension, train a small model, reverse-engineer a binary. Each of the 89 tasks drops the agent into a Docker container with a prompt and a hidden verifier, and scores a hard pass/fail — either the artifact the task asked for exists and is correct, or it doesn't. There is no partial credit and no way to talk your way to a point.

Version 2.1 is the verified iteration of 2.0: the same 89 tasks, with 26 of them repaired for bugs, timeout and resource limits, and reward-hacking robustness. It is a demanding, long-horizon suite — many tasks legitimately take an agent tens of minutes of sustained tool use — and it rewards agents that can plan, run commands, read output, and recover from their own mistakes without a human in the loop.

The Result

Here is where 80.9% sits on the current public board. The ClawCodex row is our single-run measurement inserted at its score; every other row is the leaderboard's own k=5 figure. The table shows the top of the board and ClawCodex's immediate neighbors — a few lower-ranked rows are omitted for length.

The most informative comparison on that table is the closest one on the same model tier: Claude Code on Opus 4.8 at 78.9%. ClawCodex ran the newer Opus 5, so this isn't a clean harness-only A/B — the model generation differs too, and on this same board a model bump has historically been worth far more than two points. What the row does show is that an open rebuild on a current top model lands ahead of the reference implementation's best non-Fable configuration, rather than trailing it.

How We Ran It

The whole run is reproducible from the open-source repo. Terminal-Bench is evaluated with Harbor, the eval framework behind the benchmark, and ClawCodex ships a Harbor agent adapter (eval/harbor/) that installs the CLI inside each task container and runs it headless.

Model

claude-opus-5, Anthropic's top model, at effort=xhigh (maximum reasoning budget).

Harness

ClawCodex run headless (--print --dangerously-skip-permissions), one agent per container, a 300-turn cap, and the benchmark's own per-task time limits.

Build

main at commit 95969a77 (PR #756). The run predates the v1.3.0 tag; v1.3.0 ships a handful of later fixes (#757–#764) that were not part of this benchmark.

Sampling

A single pass over all 89 tasks (k=1), two trials in parallel. No best-of-N, no task cherry-picking, no retries beyond the benchmark's own rate-limit retry.

The command is one line from the repo root, pointed at the org-qualified 2.1 dataset. It matches the run above — an OAuth Claude subscription, effort=xhigh, and the git build that registers Opus 5:

PYTHONPATH=$PWD/eval/harbor harbor run \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent clawcodex_agent:Clawcodex \
  --model anthropic/claude-opus-5 \
  --ak subscription=true \
  --ak effort=xhigh \
  --ak source=git+https://github.com/agentforce314/clawcodex@main \
  --jobs-dir eval/harbor/jobs \
  --n-concurrent 2

Why the Number Is What It Is

A rebuild's benchmark score is really a test of how faithfully it reproduces the original's behavior under pressure. When we first pointed the Harbor adapter at Terminal-Bench, ClawCodex trailed its own potential — and much of the gap wasn't the model. It was small harness infidelities that compound over a long, autonomous run: prompt guidance that had drifted from the reference, tools that were deferred a beat too aggressively, and trials lost to transient failures rather than wrong answers.

Closing it was unglamorous parity work, done in the open across a run of pull requests:

• Restored the reference's task-tool skip conditions and its "maximize parallel tool calls" guidance, which had been dropped in a prior refactor.
• Restored two dropped qualifiers in the agent's task-execution instructions and stopped an over-eager "exhaustive audit" nudge from firing on ordinary English prompts.
• Deferred only genuinely nonessential initial tools, so the agent isn't paying discovery latency it doesn't need — without hiding tools it does.
• Began hardening the run against empty model turns and transient transport drops, so an infrastructure hiccup is more likely to be retried than counted as a task failure (this work continues in the v1.3.0 release fixes).

We can't cleanly separate that harness work from the model in the leaderboard comparison, because no Claude Code / Opus 5 row exists to hold the model fixed. What we can say is narrower and still true: the same repo, harness, and every parity fix are open source, so the run is something you can reproduce rather than take on trust.

An Honest Accounting of the 17 Misses

Seventeen tasks didn't pass. Broken down, most were not the model getting the wrong answer:

8 — agent timeouts

Long-horizon builds and training jobs (a CIFAR-10 Caffe pipeline, fastText training, a Corewars tournament) that ran into the benchmark's per-task wall clock — seven scored zero, one ran out before a verdict. These are as much about pace as correctness.

1 — agent crash

One task (a MIPS interpreter build) exited non-zero and scored zero.

1 — container network error

A network failure during setup left one task with no verdict at all — not a wrong answer.

3 — empty-turn model flakes

The model returned an empty turn and the run produced nothing. This is an infrastructure-shaped failure; the transport-retry hardening in the v1.3.0 release (#757, #760) lands after this run and targets exactly this class.

4 — genuine failures

The remaining four are real: the agent produced an answer and the verifier rejected it. That's the honest floor of what's left to improve.

That breakdown is why we read 80.9% as a conservative number rather than a generous one. Here is the sharpest way to see it: 16 tasks hit the agent's time limit, and 8 of them still passed before the clock ran out — the timeout wall is a real constraint on this suite, not a sign the agent couldn't do the work. We're reporting the raw pass rate regardless, because that's the number the benchmark measures and the only one worth publishing.

What This Says About Open, Faithful Rebuilds

ClawCodex exists to be a real, runnable, multi-provider coding agent — a Python rebuild of Claude Code with a genuine tool-calling loop, not a source dump. A benchmark like Terminal-Bench is the honest test of whether "faithful rebuild" means anything: it runs the agent end-to-end, in a real shell, with no human to cover for it.

Landing around third on the public board — an open rebuild sitting in the same band as the reference implementation and the leading commercial agents, and ahead of Claude Code's Opus 4.8 and Sonnet 5 configurations — is the strongest evidence so far that the rebuild is faithful where it counts. And because the harness, the eval adapter, and every parity fix are open source, the result is something you can rerun rather than take on trust.

Conclusion

80.9% on Terminal-Bench 2.1, on Opus 5, on a single run: a top-tier open-source result that would slot around third on the public board, ahead of the reference implementation on the same model tier. It's a k=1 number against a k=5 board, so we hold it loosely — but the direction is clear, and every piece of it is in the open-source repo for anyone to reproduce.

References

• Terminal-Bench 2.1 leaderboard — tbench.ai/leaderboard/terminal-bench/2.1
• ClawCodex on GitHub — the open-source harness, Harbor adapter, and every parity fix
• Harbor — the eval framework behind Terminal-Bench