Automated, open tasks for AI on Ethereum

Experts add tasks by pull request. After review, an automated pipeline runs them on every agent and model, checks each trial, and publishes the results with their full trajectories.

The pipeline

One pull request starts everything

A new model is one line of config, so it can be tested the day it launches. A new or changed task runs on every agent and model as soon as it merges.

From pull request to websiteTask and model pull requests pass free checks with lint, oracle and nop runs, and maintainer review. After merge, CI runs new trials for every agent and model. A results pull request updates the website. All Harbor task folders also go to Hugging Face.1 · Contribute2 · Review3 · Automatic on mainAdd or edit a taska Harbor task folderAdd a modelone entry in job.yamlPR: free checkslint, oracle and nop runsMerge to maina maintainer reviewsCI runs only what's newevery agent and modelResults PRresults + trajectoriesMerge → websitethe board updatesPublish Hugging Face datasetall Harbor task foldersFrom pull request to websiteTask and model pull requests pass free checks with lint, oracle and nop runs, and maintainer review. After merge, CI runs new trials for every agent and model. A results pull request updates the website. All Harbor task folders also go to Hugging Face.1 · Contribute2 · Review3 · Automatic on mainAdd or edit a taska Harbor task folderAdd a modelone entry in job.yamlPR: free checkslint, oracle and nop runsMerge to maina maintainer reviewsCI runs only what's newevery agent and modelResults PRresults + trajectoriesMerge → websitethe board updatesPublish Hugging Face datasetall Harbor task folders

What we measure

Three pillars

Each task belongs to one of three pillars: Concepts, Build, and Act.

  1. Concepts

    Quizzes on Ethereum concepts. A bare model or an agent answers, and the verifier checks the reply.

  2. Build

    The agent writes contracts, pages, or apps, and the tests run what it wrote.

  3. Act

    The agent acts on a chain. It sends transactions, or writes a script that sends them, and the tests check what changed.

Three ways to run a task

Model only

What does the model know?

The bare model answers in one API call, with no tools or web.

Internet

Can an agent do the work?

A coding agent with a shell, the web, and its own chain when the task needs one.

Internet + Skills

Do Ethereum's own guides help?

The same agent, plus a pack of ethskills guides. The difference between the two is the skill lift.

Inside a task

A task is a folder

Each Harbor task declares its instruction, environment, verifier, and solution.

tasks/<pillar>/<name>/
  • task.toml# settings, description, and modes
  • instruction.md# the instruction the agent receives
  • environment/
  • Dockerfile# the agent's tools and starting files
  • docker-compose.yaml# optional sidecars, such as a chain
  • tests/
  • test.sh# the verifier that writes named checks
  • judge.toml# optional RewardKit questions for a judge
  • solution/# solve.sh, which the oracle runs

Chain setup is a Forge script and the tests are Forge tests, so an Ethereum developer writes them the way they already work.

Inside one trial

One task, start to finish

Take one Act task: "send tokens to the recipient." Here is what happens.

Inside one trialETH Evals reads the task folder, picks the agent, model and mode, and configures the trial. Harbor starts the task environment and its declared sidecars. The agent sends transactions to the chain. After the agent stops, the verifier reads the live chain state with Forge tests in the agent's container. RewardKit runs the judge on the host to answer yes-or-no criteria about the trajectory. ETH Evals records one result per trial and publishes results with trajectories on the website.ETH EvalsDecide what to runTask folderinstruction, environment, checks, solutionFind what hasn't runagent, model and mode with no result yetConfigure the trialinstruction, agent, checks, containersHarbor runs itstarts the containers, records the trajectorysends transactionsreads the chain statetrajectoryjudge outputcheckscontainerAgentClaude Code, Codex or OpenCodeinstructionenvironment+ Ethereum skills, in skills modecontainerChaina fresh chain or a copy of mainnetVerifierForge testsin the agent's containerLLM as judgeanswers yes-or-no questionsRewardKit on the hostRecord itOne result per trialResults + trajectories on the websiteInside one trialETH Evals reads the task folder, picks the agent, model and mode, and configures the trial. Harbor starts the task environment and its declared sidecars. The agent sends transactions to the chain. After the agent stops, the verifier reads the live chain state with Forge tests in the agent's container. RewardKit runs the judge on the host to answer yes-or-no criteria about the trajectory. ETH Evals records one result per trial and publishes results with trajectories on the website.ETH EvalsDecide what to runTask folderinstruction, environment, checks, solutionFind what hasn't runagent, model and mode with no result yetConfigure the trialinstruction, agent, checks, containersHarbor runs itstarts the containers, records the trajectorysends transactionstrajectoryreads the chain statejudge outputcheckscontainerAgentClaude Code, Codex or OpenCodeinstructionenvironment+ Ethereum skills, in skills modecontainerChaina fresh chain or a copy of mainnetVerifierForge testsin the agent's containerLLM as judgeanswers yes-or-no questionsRewardKit on the hostRecord itOne result per trialResults + trajectorieson the website

Rules every trial follows

  • Tests arrive after the agent stops. Harbor starts a separate verifier, or checks a live sidecar in the agent's container.
  • No shortcuts on the chain. Chain controls, like setting a balance by hand, are blocked by the RPC filter.
  • Judging starts after the agent stops. The verifier and the LLM judge only look once the agent is done.
  • Results stay tied to what made them. Change the task, the model or the agent, and it runs again.

Have an Ethereum task AI should handle? Add it as a task.

Read the guide →