Skip to content

Overview

ASI-Bench evaluates how far Agent×Model systems can conduct project-level scientific research as human methodological guidance is progressively withdrawn. It jointly measures general intelligence, innovation, and autonomous execution.

Why This Benchmark

Many scientific benchmarks primarily test whether a system can produce a correct answer or complete a task under extensive human guidance. ASI-Bench targets a different setting: long-horizon, project-level research in which the system must turn scientific decisions into verifiable results.

The benchmark is designed to test whether an agent can:

  • understand a scientific objective
  • inspect provided data and decide what matters
  • choose an appropriate method
  • implement and debug the workflow
  • produce output artifacts that can be checked objectively

This measures sustained end-to-end investigation rather than a single-prompt exam or the execution of a fixed procedure.

Current Benchmark at a Glance

  • 60 project-level research tasks in the first release, spanning 11 scientific domains
  • more than 1,300 candidate research ideas, 1,100 review assignments, and 2,000 task revisions during construction
  • more than 31,000 human-hours and 1,500 sandbox runs invested in review and validation
  • more than 2,600 interaction turns, 2,400 execution steps, and 35 hours of agent execution across the retained tasks

All retained tasks undergo expert review, AI-assisted auditing, sandbox execution, and scorer validation. The benchmark is community-driven and is intended to grow beyond the first release.

What Project-Level Means

In this benchmark, a task is not only a prompt. A task includes metadata, runtime requirements, data or instance generation, expected artifacts, scoring rules, and public-safe summaries for website rendering.

A successful agent run should leave behind evidence:

  • generated data files or figures
  • implementation code
  • structured answer artifacts
  • scoreable outputs
  • run metadata and provenance

That evidence is important because the website is intended to support official leaderboard results, not just informal model comparisons.

Benchmark family Typical focus ASI-Bench difference
QA-style science benchmarks answer correctness evaluates multi-step scientific workflows and artifacts
coding benchmarks code generation or issue repair adds scientific method selection and domain interpretation
ScienceAgentBench / AstaBench broad science-agent evaluation emphasizes project-level tasks with B1-B4 prompt-level control
SkillsBench-style agent benchmarks agent/tool skill use uses scientific tasks and reproducible scoring as the primary surface

The benchmark therefore evaluates a model–harness system as a research system, not the backbone model in isolation.

B1-B4 Prompt Levels

ASI-Bench evaluates the same scientific objective under four prompt levels:

  • B1: scientific background, method, equations, and procedural guidance are provided.
  • B2: the intended methodological approach and relevant constraints are provided, while implementation decisions remain with the agent.
  • B3: only the objective, available inputs, constraints, and required outputs are provided; the agent selects the strategy and method.
  • B4: the complete B3 task is retained and factually correct but non-essential information is added to test robustness to distraction.

The goal is to measure how much scaffolding an agent needs before it can solve a scientific workflow reliably.

B1-B4 Prompt Ladder

Same task goal, same data, same evaluation; guidance decreases as autonomy increases.

B1

Full procedure

Background, method, equations, and procedural guidance are supplied.

B2

Method specified

The methodological approach and constraints remain; implementation decisions do not.

B3

Method autonomous

The objective, inputs, constraints, and outputs remain; the agent determines how to solve the task.

B4

Distractor robustness

B3 remains complete while correct but non-essential context is added.

Only methodological information changes: the objective, data, required outputs, and evaluation criteria remain fixed.

What the Current Evaluation Shows

Across 18 Agent×Model configurations, the all-system mean falls from 50.91 at B1 to 29.10 at B2, 26.62 at B3, and 26.99 at B4. The largest drop is B1→B2 (−21.82), while removing the method itself from B2→B3 causes a further 2.48-point decline and B4 changes little from B3.

The main observed bottleneck is therefore method operationalization: current systems struggle more to turn a specified scientific method into a complete, validated research procedure than to select the method or ignore non-essential context.

Task Lifecycle

The public website is generated from structured benchmark metadata. In the long term, each task page should expose:

  • task title and identifier
  • domain and subdomain
  • public summary
  • expected output types
  • high-level evaluation summary
  • runtime and sandbox notes
  • safe prompt excerpt

The current public catalog is intentionally scoped to tasks that are ready to be shown on the benchmark site.

Evaluation Workflow

Compact workflow overview from task definition through result reporting.
Benchmark workflow from task definition to scoreable outputs and website-ready reporting artifacts.

At a high level, the evaluation loop is:

  1. Read the task prompt and inspect the provided data.
  2. Decide on an appropriate scientific or computational method.
  3. Implement the solution and generate required artifacts.
  4. Produce structured outputs such as data files, figures, and code.
  5. Submit the produced artifacts for official evaluation.
  6. A maintainer scores the confirmed run against the private reference material and records the result for reporting.

External runners use asibench run --no-score and cannot compute an official score locally. Reference answers and task-specific scoring configuration remain private; self-reported scores are not used for the official leaderboard.

Examples of result artifacts include:

  • run_metadata.json for run-level provenance
  • per-instance result JSON files
  • batch_overview.json
  • task_scoreboard.json
  • task_level_long.json

Reproducibility and Contamination Control

The benchmark design emphasizes:

  • sandboxed execution modes
  • explicit runtime requirements
  • structured output contracts
  • parameterized ground-truth generation
  • provenance attached to saved results

Parameterized generation helps reduce contamination risk and makes it easier to produce multiple scoreable instances without hand-authoring every case.

Leaderboard Readiness

The official leaderboard should eventually show reviewed runs accepted by benchmark maintainers. Each official result should include:

  • agent or harness name
  • model name
  • overall score
  • B1-B4 breakdown
  • task coverage
  • benchmark version
  • evaluation date
  • source or trace links when available
  • trust label such as official, reproduced, community, or unverified

This separation matters because official baselines and community submissions should not be mixed without clear provenance.

Community

Join the ASI-Bench community to discuss evaluation results, get help running the benchmark, and connect with task contributors.

Discord

Join the ASI-Bench Discord

Ask evaluation questions, discuss published results, and follow benchmark updates with the wider community.

Join Discord
WeChat

Join the ASI-Bench WeChat group

Scan the group QR code for Chinese-language updates and contributor discussion.

Show WeChat QR code
QR code for the ASI-Bench WeChat group Open WeChat and scan this code to join. If the code has expired, ask for an updated invite in Discord.