Agent harness benchmarks

7 tools

What each agent harness costs you before the model does any work: how long until you can type, how much memory it holds while it waits, and how much disk it needs. Same machine, fresh install, no login, every run interleaved across tools.

launch → ready to type lower is better
  1. agentty47 ms
  2. jcode56 ms1.2×
  3. Codex CLI402 ms8.6×
  4. Claude Code635 ms14×
  5. Aider1.0 s21×
  6. Gemini CLI3.0 s65×
  7. opencode5.2 s110×
memory while idle lower is better
  1. agentty21 MB
  2. Aider74 MB3.5×
  3. jcode106 MB5.0×
  4. Claude Code152 MB7.2×
  5. Codex CLI152 MB7.2×
  6. Gemini CLI441 MB21×
  7. opencode651 MB31×
on disk, with runtime lower is better
  1. agentty23 MB
  2. jcode163 MB7.1×
  3. Gemini CLI197 MB8.6×
  4. Claude Code325 MB14×
  5. opencode409 MB18×
  6. Codex CLI536 MB23×
  7. Aider653 MB28×
--version startup lower is better
  1. agentty2.6 ms
  2. Claude Code14 ms5.5×
  3. jcode17 ms6.6×
  4. Codex CLI77 ms30×
  5. opencode777 ms299×
  6. Aider1.1 s412×
  7. Gemini CLI2.6 s995×

median of 10 launches · same machine · fresh install, no login · method & all numbers

All numbers

toolversionready to typeidle memorydisk--version
agentty0.9.510.9.5147.0 ms 26–74 ms21.2 MB23 MB2.6 ms 2.3–4.2 ms
Claude Code2.1.3192.1.319634.8 ms 594–895 ms151.6 MB324.7 MB14.4 ms 13–26 ms
Codex CLI0.167.00.167.0401.9 ms 375–452 ms151.7 MB536.2 MB incl. 99 MB node76.9 ms 70–175 ms
Gemini CLI0.67.00.67.03.04 s 2.94–4.48 s440.6 MB196.9 MB incl. 99 MB node2.59 s 2.45–4.34 s
opencode1.19.51.19.55.19 s 4.93–6.84 s651 MB408.9 MB776.7 ms 0.74–1.37 s
jcodev0.99.1v0.99.155.6 ms 36–60 ms105.8 MB163.4 MB17.2 ms 16–37 ms
Aider0.86.20.86.21.00 s 0.95–1.04 s73.7 MB653 MB1.07 s 1.04–1.67 s

Median, with the 10th–90th percentile range under it.

Method

  • Ready to type: launch the interactive UI in a real pseudo-terminal and time until its first prompt is fully on screen, read from the rendered terminal (not raw output). Each tool first completes its one-time setup, so this is the launch a returning user sees. 10 timed launches.
  • Idle memory: resident memory of the whole process tree, 2 s after the prompt appears. jcode runs its UI as a client of a background server that it starts on first launch and leaves running; that server is counted in its memory, and it is already running for the timed launches, as it would be for a returning user.
  • Disk: everything the tool needs, as installed by its official method, including the Node.js runtime when it needs one. Aider's figure includes the Python 3.12 its installer sets up.
  • --version startup: process start to exit, 3 warm-ups then 30 timed runs.
  • Fairness: a fresh, empty home directory per tool; no accounts, keys or network needed; runs interleaved round-robin so background load on the machine lands on every tool equally. Network access doesn't change the results.
  • Close results: agentty and jcode launch within the same range (see the brackets in the table); treat that as a tie, not a win.
  • Not measured: how well each agent solves tasks. That depends mostly on the model, and every harness here can use strong ones.

Machine

Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (4 threads), 16 GB RAM, Linux 7.2.9-zen1-1-zen. A modest laptop-class CPU on purpose: startup costs that hide on a fast desktop show up here.

Reproduce it

The script is scripts/bench-agents.mjs. Install each tool by its official method, then run it; it writes the JSON this page is built from.

sh
node scripts/bench-agents.mjs ~/bench-agents