CommerceAgentBench is now public, and it is the most interesting thing anyone has shipped for agent evaluation this month. Accio, the commerce agent team inside Alibaba International, has put its internal benchmark on GitHub under an open licence: 107 tasks, fourteen offline replicas of real business software, a fresh container per task, and a leaderboard covering thirteen model families across three separate agent harnesses.
The pitch is simple and unusually honest. Most agent benchmarks ask a model questions about work. CommerceAgentBench makes the agent do the work — publish the listing, book the freight, edit the storefront theme, produce the spreadsheet — and then a verifier inspects what actually changed in the mock service afterwards. Nothing is scored on the transcript. Everything is scored on state.
That design is why the numbers are low. The best model on the best harness clears 66 of 107 tasks. Forty-one tasks are unsolved by anything currently on the board. If you have been told that autonomous agents are ready to run commerce operations, this repository is the most specific counter-argument available, and it was published by a company whose entire product line is commerce agents.
This piece covers what CommerceAgentBench measures, how the grading works, what the leaderboard says once you read past the top row, what “open source” covers and what it quietly does not, why Accio built it, and what any of it is worth if you are the one deciding whether to buy an agent this quarter. Every figure comes from the repository or its live leaderboard, and where the framing deserves a caveat, this article says so.
Table of contents
- What CommerceAgentBench Actually Measures
- How the Grading Works
- The CommerceAgentBench Leaderboard, Read Properly
- What the Headline Numbers Do Not Tell You
- What “Open Source” Covers Here
- Why Accio Built This
- How CommerceAgentBench Compares on the Benchmark Shelf
- Running CommerceAgentBench Yourself
- What This Means If You Are Buying Agents
- References
What CommerceAgentBench Actually Measures
CommerceAgentBench describes itself as a stateful agent benchmark for long-horizon commerce and business workflows. Strip the phrasing and it means three things: the tasks are long, the environment remembers what the agent did, and the grade comes from the environment rather than the answer.
The 107 tasks and how they split
The suite is fixed at 107 tasks for release v1.3.1, spread across four interface types. The split is heavily weighted toward command-line work, which tells you something about where the team thinks the failures are.
| Interface | Tasks | Share of suite | Example platforms in the repo |
|---|---|---|---|
| CLI | 53 | 49.5% | Stripe, Jira, Notion, Todoist, Amazon, Google Workspace |
| Browser | 28 | 26.2% | Alibaba, Shopify, Freightos, Reddit, Gmail, web research |
| File | 16 | 15.0% | Commerce ops, logistics, Google Trends |
| API / MCP | 10 | 9.3% | Amazon, Gmail, Google Docs, Slack, travel |
There is a second cut of the same 107. CommerceAgentBench also publishes three capability slices: 65 text-only tasks, 20 that need a browser but only text, and 22 that require vision. That last slice matters if you are evaluating a text-only deployment, because roughly a fifth of the benchmark is simply unavailable to you.
The vision slice is the one place where classic computer vision capability shows up as a hard gate rather than a nice-to-have. An agent that cannot see a rendered storefront cannot verify that it changed the theme correctly.
Fourteen replicas, not screenshots
The interesting engineering in CommerceAgentBench is the mock layer. The repository registers fourteen offline replica services: an Alibaba product publishing form, two Shopify surfaces (admin and online store), Amazon’s selling-partner API, Freightos freight booking, Gmail, Google Docs and a Google Workspace CLI, Stripe, Jira, Notion, Todoist, Reddit and a document workspace CLI.
These are not recorded screenshots. They hold state, they reject malformed input, and they run offline, which is the whole point — you do not need production accounts, and a task that fails does not fail because a live vendor changed their UI overnight. A separate showcase site renders 104 pages across eight of the UI mocks so you can see what the agent sees.
Renamed from RealReplicaBench in August
If you went looking for this project earlier in the month and could not find it, that is because the name changed. CommerceAgentBench was previously RealReplicaBench; the rebrand commits landed on 18 and 19 August 2026, and the GitHub repository URL itself was renamed on 24 August. The stated reason is scope — “Commerce Agent Bench” better describes what the suite is about and leaves room to expand.
Note the styling: the public display name is spaced as three words, while the repository, the citation key and every URL use CommerceAgentBench as one. Both refer to the same thing.
How the Grading Works
The grading design is the part worth copying even if you never run CommerceAgentBench itself. It is stricter than most published agent evaluations and it is explicit about what it does not prove.
A fresh container for every task
Every task runs in its own container, launched clean. The task tree the agent can see contains only task.md and a workspace/ directory. The graders, rubrics, private seeds, service launchers and mock source code all live outside it. Final artefacts are written to /task/outputs/.
When the agent exits, a host-side verifier reads those outputs and the isolated mock state, writes the reward record, archives the logs and trajectories, removes the container, and leaves a run directory behind. That directory holds run.yaml, summary.json, summary.md and report.html, plus a per-task folder with the manifest, the agent trace, the verifier output, the workspace outputs, screenshots and container metadata.
Deterministic graders, with a model judge where needed
Each task in CommerceAgentBench carries its own verifier, and the repository is upfront that some of them are deterministic and some are LLM-assisted. That is the honest engineering answer — you can check deterministically that a Stripe refund was issued, and you cannot check deterministically that a market research summary is any good.
It also means part of the score depends on a model’s opinion. The published results used gemini-3.1-pro-preview as the judge. That is disclosed clearly in the README, which is more than most benchmarks manage, but it is a fact worth carrying into any comparison you draw from the leaderboard.
The reproducibility contract
CommerceAgentBench pins four things and asks you to report the rest. It is a short table and it is the most useful page in the repository.
| Pinned by the contract | v1.3.1 value | You must report |
|---|---|---|
| Task set | domain_v1_all — 107 task IDs | Provider and endpoint class |
| Task definitions | This release, including workspaces and graders | Exact model and judge identifiers |
| Harness | The OpenClaw runner in the repository | Reasoning configuration |
| Runtime | A pinned accio_bench image digest | Task count, retry policy, aggregation rule |
The contract also carries a warning that deserves repeating outside this repository: compare results only within one benchmark version, and never by displayed model name alone, because routing, model snapshots, prompt adapters, retry policies and judge endpoints all change outcomes.
The CommerceAgentBench Leaderboard, Read Properly
Thirteen model families were run on each of three harnesses — Pi, OpenClaw and Accio — with the same thirteen in all three, so the rows compare directly. The live leaderboard is the source of record; the repository tables are a snapshot.
Claude Opus 5 leads all three boards
Anthropic’s Claude Opus 5 finishes first on every harness: 66/107 on Accio, 65/107 on Pi, 60/107 on OpenClaw. Here is the top of the Accio board, which is the highest-scoring of the three.
| Model | Pass | Avg. steps | Avg. time | Avg. tokens |
|---|---|---|---|---|
| Claude Opus 5 | 66/107 (61.7%) | 63.2 | 10.1 min | 3.69M |
| Claude Opus 4.8 | 59/107 (55.1%) | 67.4 | 11.6 min | 4.82M |
| Claude Opus 4.7 | 56/107 (52.3%) | 61.5 | 6.4 min | 4.32M |
| Qwen 3.8 Max | 56/107 (52.3%) | 71.2 | 15.7 min | 3.39M |
| GPT-5.6 Sol | 55/107 (51.4%) | 53.0 | 5.5 min | 1.85M |
| DeepSeek V4 Pro | 54/107 (50.5%) | 70.2 | 14.4 min | 4.50M |
| Gemini 3.6 Flash | 50/107 (46.7%) | 47.7 | 4.6 min | 2.62M |
Read down the pass column and the spread is narrow. Seventh place is 15 percentage points behind first. On a benchmark this hard, that is a crowded field rather than a runaway.
The harness moves the score as much as the model
This is the finding most people will skip, and it is the one that matters commercially. The same model scores materially differently depending on which agent harness is driving it.
Average pass rate on CommerceAgentBench v1.3.1, per harness, for five models:
GPT-5.6 Luna swings from 36/107 on OpenClaw to 48/107 on Accio — twelve tasks, or 11.3 percentage points, purely from the scaffold around the model. Qwen 3.8 Max moves from 47 to 56 across the same two harnesses. Nine tasks.
Where the harness barely matters at all
The counter-example is just as instructive. Gemini 3 Flash scores exactly 31/107 (29.0%) on all three harnesses — identical to the task. Below a certain capability threshold the scaffold stops rescuing you, which is a cleaner statement of the “agent framework won’t fix your model” argument than most vendors will make out loud.
| Model | Pi | OpenClaw | Accio | Spread |
|---|---|---|---|---|
| Claude Opus 5 | 65 | 60 | 66 | 6 tasks |
| GPT-5.6 Sol | 52 | 53 | 55 | 3 tasks |
| Qwen 3.8 Max | 53 | 47 | 56 | 9 tasks |
| GPT-5.6 Luna | 42 | 36 | 48 | 12 tasks |
| Gemini 3 Flash | 31 | 31 | 31 | 0 tasks |
What the Headline Numbers Do Not Tell You
CommerceAgentBench is better documented than most of its peers, and that documentation is exactly where the caveats are found. None of the following is hidden. All of it is easy to miss.
The published scores did not come from the public path
The README states that the published results were produced through Accio-managed evaluation endpoints, while the public path in the repository uses bring-your-own credentials. Those are not the same route. Run CommerceAgentBench yourself with your own keys and you should expect your numbers to differ from the board, possibly by more than the gap between adjacent rows on it.
That is not a scandal — running thirteen model families three times over is expensive, and someone had to pay for it. It does mean the leaderboard is a vendor-run result set rather than a community-reproduced one, and the distinction should survive into however you cite it.
Reasoning effort is not matched across vendors
Every model was run with thinking enabled at its provider’s default reasoning effort. The README says plainly that this default differs by vendor. So the board compares models as their makers ship them, not models at equal compute — a defensible choice, and a very different claim from “model A reasons better than model B”.
The judge and the home harness
Two structural points are worth stating without overclaiming either. First, the LLM-assisted verifiers were judged by a Google preview model while three Google models sit on the board being judged. Second, the Accio harness — built by the same team that wrote the tasks and the graders — produces the highest score for the leading model, and the largest gains for several others.
Neither observation proves anything. Both are the kind of thing an independent replication exists to settle, and no independent replication has been published yet.
Nobody is close to finished
The headline framing on any benchmark launch is who won. The more useful reading of CommerceAgentBench is what is left.
Every one of those bars is the best available model failing. The floor is worse: Gemini 3 Flash leaves 76 of 107 tasks unfinished on every harness it was run on.
What "Open Source" Covers Here
CommerceAgentBench is genuinely open, and the licensing is more carefully drawn than the phrase usually implies. It is worth reading before you build anything on top of it.
Two licences, one repository
| Scope | Licence | Practical effect |
|---|---|---|
| Harness, Python package, mock-service code, scripts, configs | Apache 2.0 | Commercial use allowed; keep notices, state changes |
| Task suite: definitions, workspaces, graders, rubrics | CC BY 4.0 | Reuse with attribution, including commercially |
| Mirrored stylesheets, webfonts, icons, recorded API responses | Third-party rights retained | Check THIRD_PARTY_NOTICES.md before redistributing |
| The names “Accio” and “Commerce Agent Bench” | No trademark grant | A fork needs its own name |
That third row is the one to actually action. The mock services look like real products because they mirror real assets, and those assets did not become Apache-licensed by being copied into an Apache-licensed repository.
What you cannot reproduce from the repository alone
The harness, the tasks and the graders are all there. The published scores are not reproducible from them, because the endpoints that generated those scores were Accio-managed. You can reproduce the method exactly. You cannot reproduce the board exactly. Both statements can be true at once and the repository says so.
Reception so far
CommerceAgentBench went public on 2 August 2026 and had gathered roughly 1,200 GitHub stars and 83 forks by the end of the month, with a single open issue. There is no accompanying paper yet — the citation block asks you to cite the repository and the exact commit until one is published — and the commit history is essentially one maintainer. For a benchmark asking to be taken as a standard, that is early days.
Why Accio Built This
Accio is not a research lab that happened to write a benchmark. It is a commerce agent product, and CommerceAgentBench is the measurement layer underneath it.
The product behind the benchmark
Alibaba International launched Accio Agent in August 2025, pitched as the first AI agent for global trade, and followed it with Accio Work on 23 March 2026 — an enterprise agent workspace aimed at sourcing, market research, storefront launches and operations. Our earlier write-up of Alibaba Accio Work covers what that product promises to smaller businesses.
CommerceAgentBench is the natural consequence. If you are shipping autonomous AI agents that book freight and publish listings, you need a way to know whether a model change made them better or worse, and no public benchmark measured that.
The skills repo alongside it
The same team publishes an open skills collection under the AccioWork organisation — product selection, supplier sourcing, market insight, Shopify dropshipping, product descriptions, review summarising, tariff calculation, SEO optimisation and e-commerce marketing — written to the OpenClaw and MCP standards so they run in other agent frameworks. Reading the skills next to the benchmark tasks tells you plainly which workflows Accio thinks are commercially live.
And a sibling benchmark
Accio also maintains Business Arena, which asks a different question: can an agent run a seller business over a 30-day market horizon, handling sourcing, pricing, advertising, inventory, negotiation and compliance as demand shifts? Its scoreboard is denominated in money rather than pass rates, and its published snapshot covers 150 completed runs across 15 model families, led by Gemini 3.1 Pro at $188,488 mean final worth.
The pairing is deliberate. CommerceAgentBench measures whether an agent can execute a defined task correctly; Business Arena measures whether it can make sensible decisions when nobody has defined the task at all.
The contribution ask
The repository is openly recruiting mock environments, and the reasoning is the sharpest paragraph in it: a fixed task set decays, because models saturate it and its answers drift into training data. Each new offline replica — a real service’s API semantics, state transitions and, above all, its rejections — is a family of tasks no model has trained on. Contributed services land in a contrib directory outside the scored set, then get promoted into a later release.
How CommerceAgentBench Compares on the Benchmark Shelf
Agent evaluation has fragmented into several incompatible traditions, and it helps to know which one you are reading before you quote a number from it.
Three kinds of agent benchmark
| Style | What it scores | Failure it misses |
|---|---|---|
| Static question sets | The final answer string | Everything about execution, tools and state |
| Live-service agent tests | Real outcomes on real accounts | Nothing — but results drift and cannot be replayed |
| Stateful replicas (CommerceAgentBench) | Verified state change offline | Whatever the replica simplifies away from production |
The third row is the honest limitation of the whole approach. A mock Shopify admin is not Shopify. It is a fixed, inspectable approximation, and an agent could learn the approximation’s quirks rather than the real product’s.
Why offline replicas still win on balance
Against that, the alternative is worse. A benchmark run against live services cannot be replayed six months later, cannot be run by a reviewer without credentials, and quietly changes its own difficulty every time a vendor ships a UI update. CommerceAgentBench trades a little realism for the ability to say two runs measured the same thing.
The saturation clock is already ticking
Every fixed benchmark has a shelf life, and this one is public now, which means its tasks will start appearing in training corpora. The team says as much in the contributing guide. Treat the current board as a snapshot with a decay date rather than a permanent ranking, and note that Accio has already told you how it intends to refresh it.
Running CommerceAgentBench Yourself
The barrier to entry is low in software terms and not low at all in tokens.
What you need
Docker on linux/amd64, Python 3.11 or newer, and API keys for both your model provider and your judge provider. The repository ships a pinned runtime image digest so your container matches the reference one. The public API routes have been exercised against local protocol recorders, which proves request construction and response parsing — not that your account has entitlement, quota or billing to actually run them.
Budget for the token bill
Average token consumption per task is published for every model, and it is the number most teams will not have planned for.
Multiply the top bar by 107 tasks and a single full sweep of CommerceAgentBench costs over half a billion tokens. GPT-5.5 does the same suite for roughly a third of that while scoring 48/107 against Claude Opus 4.8’s 59 — which is a real trade-off, not a rounding error, and one no single pass-rate column will show you.
Start with one task
The repository’s own advice is to run with --limit 1 and your own keys first, recording the provider and model snapshot in the run metadata, before committing to a full sweep. Take it. The cheapest way to discover that your judge endpoint is misconfigured is on task one, not task 107.
What This Means If You Are Buying Agents
CommerceAgentBench is a research artefact, but there are three decisions it should change for anyone evaluating commercial agent products this quarter.
Ask which harness the vendor benchmarked on
The twelve-task swing on GPT-5.6 Luna is the practical headline. When a vendor quotes you a pass rate, the model name is half the answer; the scaffold driving it is the other half. Ask for both, and ask on which version of the benchmark.
Treat 60% as the state of the art, not the ceiling of your risk
The best publicly measured agent completes under two thirds of a curated commerce task suite built by people who wanted it to be completable. Any deployment plan for artificial intelligence in a revenue-carrying workflow needs a human review path for the other third. Our AI models, tools and releases hub tracks how quickly that number is moving.
Build your own replica before you trust anyone’s score
The most transferable idea in CommerceAgentBench is not the leaderboard, it is the method: replicate the system your agent will touch, seed it with state, define the state change that counts as success, and grade the environment rather than the transcript. That works on your own internal tooling, at your own scale, and it answers the only question that actually matters — whether the agent can do your job, not an average of someone else’s.
References
Commerce Agent Bench live leaderboard
Commerce Agent Bench mock service showcase
AccioWork open-source agent skills
Alibaba International Launches Accio Work, an Enterprise AI Agent for Global Businesses
Alibaba International Releases the World’s First AI Agent for Global Trade
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.