At the Dawn of Artificial Superintelligence

ASI-Bench tests how far AI can conduct project-level scientific research as human methodological guidance is progressively withdrawn.

Current leaderboard Latest aggregate results

Task set: — · B1–B4 · — rounds

—ranked
Loading benchmark scope…

Scientific breadth

Scientific Domains and Public Tasks

ASI-Bench covers a wide spectrum of computational science workflows.

Task registry

Featured Public Tasks

The cards below are selected from the generated public catalog and spread across domains when possible.

Get started in 60 seconds

Quick Start

Install the public runner, produce artifacts with your agent, and submit them for official scoring.

1 
pip install asibench
asibench task pull --output-dir hf_instances/

Install the runner and download public prompts plus input data.

2 
asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns --output-dir out/

Run your agent in produce-only mode and collect its declared outputs.

3 
asibench submit --results-dir out/

Upload the run, review the parsed summary, and confirm it for scoring.

Open the complete quick start →

Evaluation pipeline

From Task to Auditable Result

ASI-Bench combines scientific tasks, B1-B4 prompt levels, agent harnesses, and reproducible scoring into one auditable evaluation stack.

01

Scientific workflow task

Parameterized AI4Sci task with data, expected artifacts, runtime requirements, and scoring rules.

02

B1-B4 prompt level

Same scientific goal under decreasing guidance, from execution support to autonomous problem solving.

03

Agent / harness + model

CLI agent, scaffold, or direct baseline paired with a specific model and run configuration.

04

Sandbox, scorer, and report

Structured artifacts become task scores, B-level breakdowns, and reviewed leaderboard entries.

Evaluation pipeline from task definition through result reporting.

Why it matters

General Intelligence, Innovation, and Autonomous Execution

Project-level research

Tasks span problem understanding, method selection, implementation, experimentation, failure diagnosis, refinement, and validation.

Cross-domain generality

The first release contains 60 research tasks across 11 scientific domains, testing transfer across distinct data, methods, and validation criteria.

B1-B4 guidance gradient

The objective, data, outputs, and scoring stay fixed while methodological guidance is progressively removed and distractor robustness is tested.

Verifiable execution

Agents must turn scientific decisions into reproducible artifacts that pass sandbox execution and task-specific scoring.

Updates

Release Notes

  • 2026-08-16 Claude Code + GLM-5.3 at Max effort added; the leaderboard now contains 18 supplied aggregate combinations.
  • 2026-08-04 Task catalog aligned to the 60-task ASI-Bench seed42 benchmark set across 11 domains, with examples and non-benchmark definitions excluded.
  • 2026-04-22 Homepage, catalog, and leaderboard became data-driven from generated benchmark JSON.
  • 2026-04-22 Benchmark figures added: domain coverage, B1-B4 prompt ladder, and evaluation workflow.