Skip to content

Benchmark

Terminal-Bench

Terminal-Bench is a benchmark that evaluates how well AI agents complete real-world tasks via command-line interactions in a terminal environment.

Known aliases

  • Terminal Bench
  • Terminal-Bench 2
  • terminal-bench-2
  • Terminal-Bench 2.0
  • Terminal Bench 2.1
  • Terminal-Bench 2.1
  • Terminal-Bench 3.0
  • TerminalBench 3.0
  • Terminal Bench 4
  • Terminal-Bench 4
  • Terminal Bench 4.0
  • Terminal-Bench 4.0
  • Terminal-Bench Hard
  • Terminal Bench v2.1
  • Terminal-Bench v2.1

Current stories

buildConfirmed13 publishers

Muse Spark Open's license asks hosted-model sellers above $20M in revenue for a written deal with Meta

Meta released Muse Spark Open, an 816B-parameter open-weight model that activates 42B parameters per token. Per The New Stack, a company above $20M in revenue needs a separate Meta agreement before selling it as a hosted service.

Perspective Coverage

13 publishers
Builder
Builder 40%
Operator
Operator 32%
Investor
Investor 28%

Reality

Evidence55
Adoption30
Hype gap+35
Incentives65
Confidence65
buildOne report1 publisher

Haiku 5.5's 100K-token cliff cuts its price advantage over Sonnet 5.5 from 20x to 4x

Anthropic's Claude Haiku 5.5 costs $0.10/$0.50 per million tokens up to 100,000 input tokens, then $0.50/$2.50 on the whole request. A dev.to breakdown advises sending Haiku only narrow, verifiable subagent work and keeping Sonnet 5.5 as the default coder.

Publishers:dev.to

Reality

Evidence58
Adoption
Not scored
Hype gap+35
Incentives
Not scored
Confidence60
buildOne report1 publisher

Claude Sonnet 5.5 outscores Opus 5.5 on Terminal-Bench 4.0, according to a dev.to post

Anthropic's Claude Sonnet 5.5 resolves 70.6% of Terminal-Bench 4.0 tasks against 66.4% for Opus 5.5, a dev.to post reports. If that gap holds on a team's own tasks, the mid-tier model is the one to test first as the default for terminal agents.

Publishers:dev.to

Reality

Evidence20
Adoption
Not scored
Hype gap+45
Incentives
Not scored
Confidence25
buildConfirmed22 publishers

Prompt length sets what Anthropic's Haiku 5.5 price cut is worth to each workload

Anthropic launched Claude Haiku 5.5 at an average price about 75% below Haiku 4.5. The saving varies widely with prompt length, so teams moving classification, support or query traffic need to price their own requests before they switch models.

Perspective Coverage

22 publishers
Builder
Builder 47%
Operator
Operator 36%
Investor
Investor 17%

Reality

Evidence66
Adoption30
Hype gap+25
Incentives70
Confidence65
buildConfirmed8 publishers

Cursor's task-cost table shrinks Grok 4.7's 7.5x list-price gap to about 1.9x

SpaceXAI lists Grok 4.7 at $2 and $6 per million tokens against $10 and $50 for GPT-6 Astra and Claude Fable 5.1, and the only per-task figures in the record put a Cursor run at $4.69 against about $9.

Perspective Coverage

8 publishers
Builder
Builder 51%
Operator
Operator 32%
Investor
Investor 17%

Reality

Evidence60
Adoption30
Hype gap+25
Incentives65
Confidence58
buildConfirmed6 publishers

DeepSeek's V4.1-Flash report prices a live agent's memory at 890 bytes per token

The quarter-size global KV cache and eighth-size persistent storage are serving-cost claims rather than benchmark scores, and banking the second one requires a prefix-cache tier your stack has to already run.

Perspective Coverage

6 publishers
Builder
Builder 53%
Operator
Operator 33%
Investor
Investor 14%

Reality

Evidence66
Adoption42
Hype gap+14
Incentives64
Confidence63
buildOne report1 publisher

Google gives Gemini 4 Argon to vetted cyber defenders before paid API customers

Google DeepMind's Gemini 4 Argon, priced at half of Claude Opus 5.5, goes first to vetted cyber defenders, and paid API customers have no date. Until Google sets one, teams can only check its benchmark claims on paper.

Publishers:dev.to

Reality

Evidence35
Adoption15
Hype gap+30
Incentives65
Confidence35
buildOne report1 publisher

Failed runs decide whether Claude Sonnet 5.5 undercuts Opus 5.5

Anthropic prices Claude Sonnet 5.5 at half Opus 5.5's per-token rate, but Artificial Analysis measured $7.67 per task at max effort against Opus's $5.98. In The New Stack's own tests, runs that were billed but never finished decided which model cost less per completed job.

Publishers:thenewstack.io

Reality

Evidence55
Adoption
Not scored
Hype gap+20
Incentives35
Confidence50
buildConfirmed4 publishers

Post-training alone took GLM-5.3 from 4.6 to 28.3 on Terminal-Bench 3.0

Z.ai says the base model did not change between GLM-5.2 and GLM-5.3, so the coding jump and the doubled exploitation score come out of the same post-training run. Security teams inherit the second half.

Perspective Coverage

4 publishers
Builder
Builder 49%
Operator
Operator 32%
Investor
Investor 19%

Reality

Evidence50
Adoption30
Hype gap+30
Incentives70
Confidence60
buildConfirmed13 publishers

Reflection sells Beam to sovereign-AI buyers on a 3-to-4x inference compute claim

Reflection AI says Beam, its 501-billion-parameter open-weight model, needs three to four times less compute than comparable open models. Buyers have only the company's figures until the weights ship later in October.

Perspective Coverage

13 publishers
Builder
Builder 37%
Operator
Operator 35%
Investor
Investor 28%

Reality

Evidence35
Adoption8
Hype gap+40
Incentives70
Confidence55
buildOne report1 publisher

Gemini 4 Argon's per-task bill doubles when Google's launch discount ends

Google is charging $2 and $10 per million input and output tokens for Gemini 4 Argon at launch, half the $4 and $20 it will charge later. For a model that writes long answers and is open only to vetted cyber defenders, cost per task is the figure to check before switching.

Publishers:dev.to

Reality

Evidence45
Adoption15
Hype gap+20
Incentives60
Confidence40
buildOne report1 publisher

Reflection's Beam splits two coding benchmarks with GLM 5.2 before its weights ship

Reflection AI's 501-billion-parameter Beam beats GLM 5.2 on SWE-bench Pro and trails it on Terminal-Bench, by the company's own scores. Its claimed compute saving covers only part of a serving bill, so planning around it as a US-built open model has to wait for outside tests.

Publishers:runtimewire.com

Reality

Evidence35
Adoption8
Hype gap+30
Incentives75
Confidence40
buildOne report1 publisher

Rewriting scaffolded agent runs before fine-tuning lifts a 27B model to 74.2% on Terminal-Bench 2

IntelligenceLab and Maryland researchers say rewriting scaffolded agent runs lifted Qwen-3.8-27B to 74.2% on Terminal-Bench 2, versus 53.4% from raw traces. The result argues for replaying harness-assisted wins in a plain shell before training on them.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+20
Incentives
Not scored
Confidence40
buildOne report1 publisher

GLM 5.3 adds no parameters, and its sharpest jump is in finding vulnerabilities

Zhipu says every gain in GLM 5.3 came from post-training on the same ~744B MoE it shipped as GLM 5.2. The largest claimed leap, in vulnerability discovery, is also the least quantified.

Publishers:dev.to

Reality

Evidence25
Adoption10
Hype gap+35
Incentives70
Confidence35
investConfirmed5 publishers

Cached context shrinks the discount from Anthropic's half-price Sonnet 5.5

Anthropic released Sonnet 5.5 at $2 and $10 per million input and output tokens, half the Opus 5.5 rate. How much a buyer saves by moving work down a tier depends on tokens burned per task and on cache reads priced identically on both models.

Perspective Coverage

5 publishers
Builder
Builder 46%
Operator
Operator 38%
Investor
Investor 16%

Reality

Evidence55
Adoption35
Hype gap+20
Incentives70
Confidence60
buildOne report1 publisher

NVIDIA let an optimizer agent find the four token savers in its SoL-Pi harness for Pi

NVIDIA released SoL-Pi, an MIT-licensed Pi extension with four token-saving harness mechanisms that an optimizer agent found. A dev.to review puts the saving near one third on long sessions, bought with a few lost solves.

Publishers:dev.to

Reality

Evidence45
Adoption
Not scored
Hype gap+20
Incentives
Not scored
Confidence45
buildConfirmed14 publishers

One model string moves Vercel AI Gateway traffic to Claude Sonnet 5.5

Vercel's AI Gateway now routes Claude Sonnet 5.5 through a single model ID, according to a dev.to review of the week's releases. The benchmark and cost figures come only from that third-party review, so a team's own tests decide when regulated workloads move.

Perspective Coverage

14 publishers
Builder
Builder 47%
Operator
Operator 30%
Investor
Investor 23%

Reality

Evidence58
Adoption48
Hype gap+35
Incentives62
Confidence60
investOne report1 publisher

Google's open-source RRSI rewrites agent scaffolding to add six points on Terminal-Bench

Google Cloud AI Research has released RRSI, an Apache 2.0 tool whose self-rewriting agent harness lifted Terminal-Bench 2.1 scores from 74.2% to 80.2%. Any team can use it commercially, though on tasks the agent never trained against the reported gain falls to 4.7 points.

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives50
Confidence40
buildConfirmed3 publishers

Kimi K3 on its cheapest host undercuts Fireworks' Ember-1 despite a 23% cut in reasoning tokens

Fireworks' Ember-1 used 23% fewer reasoning tokens than Kimi K3 in The New Stack's tests, yet Kimi on the cheapest host would cost $1.96 to Ember's $2.48. Ember beats Fireworks' own Kimi rate and loses at the cheapest, so buyers have to price the host before the model.

Perspective Coverage

3 publishers
Builder
Builder 52%
Operator
Operator 30%
Investor
Investor 18%

Reality

Evidence55
Adoption30
Hype gap+25
Incentives70
Confidence58
buildOne report1 publisher

Claude Opus 5.5 ties Fable 5.1 on two coding tests while costing 28% to 50% less per run

Claude Opus 5.5 matched Fable 5.1 on every hidden test in two New Stack coding trials, at $0.75 and $1.42 per run against $1.50 and $1.96. It needed more tokens and more minutes to get there, so the saving holds only for tasks that resemble these.

Publishers:thenewstack.io

Reality

Evidence52
Adoption
Not scored
Hype gap+15
Incentives40
Confidence55

Earlier coverage

  1. Bedrock serves xAI's Grok 4.7 through the Converse, Responses and Chat Completions APIs

    Build · September 28, 2026 · Confirmed2 publishers

  2. Sonnet 5.5 comes within two points of Opus 5.5 at half the per-token price

    Product · September 30, 2026 · Confirmed11 publishers

  3. DeepSeek open-sources the harness, then raises the price of the model

    Build · August 15, 2026 · Confirmed4 publishers

  4. GLM-5.3 keeps GLM-5.2's base model and claims 50% more on coding: plan for shorter eval cycles

    Build · August 16, 2026 · Confirmed5 publishers

  5. Anthropic Cuts Cache-Read Prices by 75%; Cache Reads Were ~60% of a Heavy Agent's Bill Before the Cut

    Invest · September 1, 2026 · Confirmed2 publishers

  6. Gemini 3.8 Flash's introductory price doubles on December 31, 2026

    Build · September 2, 2026 · Confirmed8 publishers

  7. Spark 1.3's index jump lands on the three tests that carry half the score

    Build · September 3, 2026 · Confirmed6 publishers

  8. xAI holds Grok's $2 token price for a model 40 Elo points behind Fable 5.1

    Invest · September 21, 2026 · Confirmed3 publishers

  9. Opus 5.5 diverts most cybersecurity requests to the older Opus 4.8

    Leadership · September 22, 2026 · Confirmed2 publishers

  10. Price cuts minutes apart send agent routing back to the spreadsheet

    Product · September 22, 2026 · Confirmed5 publishers

  11. Box measured Claude Opus 5.5 using a third of the tokens Opus 5 needed

    Product · September 22, 2026 · Confirmed7 publishers

  12. Anthropic prices its newer Sonnet a third below Sonnet 4.5

    Build · September 20, 2026 · One report1 publisher