37 tasks8 agents992 trials
The Open Benchmark for AI on Ethereum
We evaluate agents used by developers, and the models behind them, on real Ethereum work. Every task, trial and trajectory is public, allowing anyone to run the same tasks.
We evaluate three pillars: Concepts, Build, and Act.
Results
The same work, with and without Ethereum skills.
| Configurationmodel · agent | ||||
|---|---|---|---|---|
 Opus 5.5Claude Code · effort high | ||||
 GPT-6 AstraCodex · effort high | ||||
| GPT-6.1 SolCodex · effort high | ||||
 Gemini 4 ArgonOpenCode · effort high | ||||
| Grok 4.7OpenCode · effort high | ||||
| DeepSeek V4.1 FlashOpenCode · effort high | ||||
| Kimi K3.1 ProOpenCode · effort high | ||||
| GLM 5.3OpenCode · effort high |
How to read it: Each row is an agent or a model, scored on three pillars: Concepts, Build, and Act. Overall is the average of the pillar scores. Click a pillar cell to see its tasks and every trial behind them.
Results by task
A detailed view of the Leaderboard. See how many trials each agent passed on every task, grouped by pillar.
37 tasks · Internet
| Task | Opus 5.5Claude Code · effort high | GPT-6 AstraCodex · effort high | GPT-6.1 SolCodex · effort high | DeepSeek V4.1 FlashOpenCode · effort high | Gemini 4 ArgonOpenCode · effort high | Kimi K3.1 ProOpenCode · effort high | Grok 4.7OpenCode · effort high | GLM 5.3OpenCode · effort high |
|---|---|---|---|---|---|---|---|---|
| Concepts25 tasks | ||||||||
| Accounts 1271 permit2 cross account replayconcepts/accounts-1271-permit2-cross-account-replay | ||||||||
| Accounts 4337 unused gas penaltyconcepts/accounts-4337-unused-gas-penalty | ||||||||
| Accounts 7739 typeddatasign domainconcepts/accounts-7739-typeddatasign-domain | ||||||||
| Agent registriesconcepts/agent-registries | ||||||||
| Compiler legacy init orderconcepts/compiler-legacy-init-order | ||||||||
| Compiler memory bytes deleteconcepts/compiler-memory-bytes-delete | ||||||||
| Compiler transient clear collisionconcepts/compiler-transient-clear-collision | ||||||||
| Compiler tuple assignment self transferconcepts/compiler-tuple-assignment-self-transfer | ||||||||
| Evm transfer refund to safeconcepts/evm-transfer-refund-to-safe | ||||||||
| Evm try catch empty returnconcepts/evm-try-catch-empty-return | ||||||||
| Forwarder multicall suffix oz5concepts/forwarder-multicall-suffix-oz5 | ||||||||
| Governance arbitrum block clockconcepts/governance-arbitrum-block-clock | ||||||||
| Lending small position close factorconcepts/lending-small-position-close-factor | ||||||||
| Permit2 7702 signatureconcepts/permit2-7702-signature | ||||||||
| Permit2 zero expirationconcepts/permit2-zero-expiration | ||||||||
| Signatures oz ecdsa signature as keyconcepts/signatures-oz-ecdsa-signature-as-key | ||||||||
| Signatures permit2 signaturetransfer frontrunconcepts/signatures-permit2-signaturetransfer-frontrun | ||||||||
| Signatures permit2 witness stringconcepts/signatures-permit2-witness-string | ||||||||
| Signatures usdc contract permitconcepts/signatures-usdc-contract-permit | ||||||||
| Smart accounts p256 precompileconcepts/smart-accounts-p256-precompile | ||||||||
| Smart accounts userop hash v08concepts/smart-accounts-userop-hash-v08 | ||||||||
| Swaps permit2 zero expirationconcepts/swaps-permit2-zero-expiration | ||||||||
| Upgrades stateless parent gapconcepts/upgrades-stateless-parent-gap | ||||||||
| Wallet atomic readyconcepts/wallet-atomic-ready | ||||||||
| Wallet batch status 600concepts/wallet-batch-status-600 | ||||||||
| Build9 tasks | ||||||||
| Aave savings pagebuild/aave-savings-page | ||||||||
| Creator coinsbuild/creator-coins | ||||||||
| Keeper depositsbuild/keeper-deposits | ||||||||
| Signed votesbuild/signed-votes | ||||||||
| Swap pagebuild/swap-page | ||||||||
| Token governorbuild/token-governor | ||||||||
| Token launchpadbuild/token-launchpad | ||||||||
| Token lending marketbuild/token-lending-market | ||||||||
| Voting pagebuild/voting-page | ||||||||
| Act3 tasks | ||||||||
| Treasury eth saleact/treasury-eth-sale | ||||||||
| Vault v2 concentration rebalanceact/vault-v2-concentration-rebalance | ||||||||
| Vault v2 market exitact/vault-v2-market-exit | ||||||||
Trials passed per task. Click a scored cell for its instruction and trials. Errors are excluded from counts.