Frame-hash replays let CodeScene's agents refactor 300,000 lines of Street Fighter III for $4,000
CodeScene's agents refactored 300,000 lines of Street Fighter III in three weeks for about $4,000 in tokens, taking its Code Health score to 10.0. The run relied on a frame-by-frame replay check and on the score the agents were told to optimize, so the promised savings on later feature work still need their own measurement.
The Engineer · Build desk
What happened
- CodeScene ran the project with researcher Marcus Borg on a decompiled version of the 25-year-old game, written in C.
- Along the way the agents wrote their own playbook of 22 recipes and 82 notes, including codebase-specific ones such as Shared Index Range and Uniform Step Table.
- The team settled on Claude Code with Opus after finding it significantly better than Codex with Sol at documenting patterns, while smaller models stalled on files.
- Daniel Webb, one of the two engineers on the project, said the work was merged to main on a fork through 54 pull requests.
Why it matters
- What it limits constraint A team whose code has no per-change behavioral check as strict as frame-hash replay would be running the same agents without the gate that made these changes safe to merge.
- What it costs cost Reviewers pay the part of the bill that the token figure leaves out: 54 pull requests averaging about 54 commits and 4,700 changed lines each.
- What doesn't add up contradiction The podcast notes put the starting score at 5.4 and InfoQ at 5.6, so even the baseline of the project's main evidence of improvement differs between the two accounts.
Next story · Build · yesterday
Codex agent solves 49 of 50 unpublished physics problems for under $3 eachConfirmed2 publishers
Street Fighter III came with its own regression oracle. The game already hashed each rendered frame and compared the hashes [4]. CodeScene's replay-trace harness compared the rollback state hash frame by frame, so behavior could be checked after every change [7]. Every agent edit got a pass or a fail inside the loop. Paolo Perrone wrote: "most refactor claims i've read rest on a green test suite, which only tells you the tests survived. replaying traces against a fighting game sets a much higher bar." [13]
The engineers who built the oracle acknowledge a gap in it. Webb, who is CTO at NeoSee, said the harness ran as a pre-commit hook and that some failures were fixed without being observed [20]. Asked whether it caught subtle frame-timing regressions, he said there might be no definitive answer [20]. Performance is also an open question. Marc Bouvier asked about framerate, memory use and input latency, and Webb said a performance specialist was being brought in [19].
The quality signal ran through the same loop. A CodeHealth MCP server gave the agents a deterministic score to optimize, and they used it to judge whether a transformation had helped [12]. The finishing 10.0 is that same score [29]. Tornhill, CodeScene's founder, calls 9.5 or higher "AI-optimal code" [1][28]. Denis Baltor argued that DRY concerns duplicated knowledge and intent, and that recipes built on loops differing only in their ranges may be treating lines that look alike as shared knowledge [16]. Asko Nõmm said architecture goes unmeasured, so code can look healthy while fundamental problems surface later [18]. One choice deserves credit. The playbook logged failed attempts, including transformations that made Code Health worse [9].
About $4,000 in tokens bought 2,903 commits and 252,055 modified lines [6][11]. That works out to roughly $1.38 a commit and 1.6 cents a modified line [23][24]. At those rates the invoice is the least interesting thing the project produced. The agents touched about 84% of the 300,000 lines [22]. Tornhill said: "I've never ever seen anything close to this before. A rewrite at that scale would have been a 12-to-18-month project. And now that can be automated for a fraction of the cost at a fraction of the time." [3]
The "few days" in the podcast notes covers the uplift alone. Preparation took about two weeks as a side project, while the team tried different models and patterns [5]. Two weeks plus a few days comes to roughly the three weeks InfoQ reports [21].
I think the refactor result holds within its harness. The payoff is a separate claim. The podcast notes say the team measured how the cleanup led to better and cheaper AI development of new features [27], but the public show notes and InfoQ's report do not include those figures. Nõmm raised a second problem. Claude Code and Codex are tuned to their own models, he argued, so it is unclear how much of a result measures the model and how much measures the harness, and cost would vary the same way [17]. Konrad Otrębski, a tech lead and consultant, wrote: "I think the real true test of AI capability here would be to offer this courtesy of refactoring to some famous open source project, say Grafana. The definition of done is ofc merging it to master." [15]
What to watch
- Release of the feature-development measurements the podcast says CodeScene took, with cost per feature before and after the cleanup.
- The performance specialist's results on framerate, memory use and input latency for the refactored game.
- An agent-driven refactor merged to master of a maintained open-source project such as Grafana, the test Otrębski proposed.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption10
- Hype gap+40
- Incentives70
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Adam Tornhill is the founder of CodeScene and author of Your Code as a Crime Scene.
- [2]
CodeScene worked with researcher Marcus Borg on a decompiled version of Street Fighter 3, a 25-year-old game with 300,000 lines of C code.
- [3]
"I've never ever seen anything close to this before. A rewrite at that scale would have been a 12-to-18-month project. And now that can be automated for a fraction of the cost at a fraction of the time."
- [4]
The game already had a test approach that computed the hash of each rendered frame and compared them, giving agents a way to verify refactoring did not break anything.
- [5]
The preparation phase took about two weeks as a side project, exploring different models and patterns; the uplift then took a few days and cost roughly $4,000 in API tokens.
- [6]
Coding agents refactored a 300,000-line C codebase over three weeks at a token cost of roughly $4,000.
- [7]
A replay-trace harness compared the rollback state hash frame by frame, so behavior could be checked after every change.
- [8]
The agents accumulated a refactoring playbook ending with 22 recipes and 82 supporting notes, including codebase-specific recipes such as Shared Index Range, Action Parameter and Uniform Step Table.
- [9]
Failed attempts were recorded in the playbook, including transformations that made Code Health worse.
- [10]
The team settled on Claude Opus, reporting Claude Code with Opus was significantly better than Codex with Sol at capturing and documenting emerging patterns; files often plateaued when smaller models ran the task.
- [11]
The work produced 2,903 commits across 726 files and modified 252,055 lines.
- [12]
The CodeHealth MCP Server gave agents a deterministic score to optimize and to judge whether a transformation had helped.
- [13]
"most refactor claims i've read rest on a green test suite, which only tells you the tests survived. replaying traces against a fighting game sets a much higher bar."
- [14]
Daniel Webb, CTO at NeoSee and one of the two engineers who did the work, said it was merged to main on a fork through 54 pull requests.
- [15]
"I think the real true test of AI capability here would be to offer this courtesy of refactoring to some famous open source project, say Grafana. The definition of done is ofc merging it to master."
- [16]
Denis Baltor argued that DRY concerns duplication of knowledge and intent, not identical lines, and that recipes defined by loops differing only in ranges or control structures differing only in the function invoked may be collapsing the two.
- [17]
Asko Nõmm argued that since Claude Code and Codex are tuned to their own models, it is unclear how much of the result measures the model and how much the harness, and that cost would vary the same way.
- [18]
Nõmm added that architecture remains unmeasured, so code can look healthy while fundamental problems surface later.
- [19]
Marc Bouvier asked whether non-functional behavior such as framerate, memory use and input latency had improved; Webb said a performance specialist was being brought in.
- [20]
Asked whether the harness caught subtle frame timing regressions, Webb said there might be no definitive answer, because it ran as a pre-commit hook and some failures were fixed without being observed.
- [21]
About two weeks of preparation plus a few days of uplift comes to roughly the three weeks InfoQ reports.
- [22]
About 84% of the codebase's lines were modified.
- [23]
Token cost was roughly $1.38 per commit.
- [24]
Token cost was about 1.6 cents per modified line.
- [25]
The 54 pull requests averaged about 54 commits each.
- [26]
The 54 pull requests averaged about 4,700 modified lines each.
- [27]
Tornhill's team refactored the Street Fighter 3 codebase, more than 300K lines, in a few days for a few thousand dollars in tokens, and measured how that led to better and cheaper AI development of new features.
- [28]
CodeScene's metrics first scored the code base 5.4 on a 10-point scale, roughly the industry average for enterprise code; Tornhill calls code scoring 9.5 or higher 'AI-optimal code'.
- [29]
The codebase moved from a Code Health score of 5.6 to 10.0.
Sources
2 independent publishers whose own reporting we read for this story.
- infoq.comAgents Refactor 300K Lines in Three Weeks, and Practitioners Ask What it Proves - InfoQ
1 article · September 29, 2026
- Refactoring Street Fighter With AI 🕹️ — with Adam Tornhill
refactoring.fm
1 article · October 2, 2026
Topics and entities
Topics
- Agentic refactoring
- Legacy Code Modernization
- Code quality metrics
Entities
- Claude Opus
- Lund University
- CodeHealth MCP Server
- Codex
- Street Fighter III: 3rd Strike
- Claude Code