How Tasks Work

Each ASI-Bench task is a compact scientific workflow with public metadata, dataset-distributed inputs, an output contract, and four benchmark prompt levels that vary how much guidance an agent receives.

Explore the benchmark Browse the task catalog → Have a scientific workflow? Contribute a task →

Public Catalog Snapshot

Task and domain counts below are generated from the public catalog rather than maintained by hand.

What an Agent Must Do

A typical task asks an agent to:

1. understand a scientific objective and inspect the supplied data
2. choose and implement an appropriate computational method
3. debug the workflow inside a controlled runtime
4. produce declared files that can be checked objectively

B1–B4 Prompt Levels

The scientific goal and evaluation stay fixed while guidance decreases:

• B1 — scientific background, method, equations, and procedural guidance are provided
• B2 — the intended methodological approach and relevant constraints are provided, while implementation decisions remain with the agent
• B3 — only the objective, available inputs, constraints, and required outputs are provided; the agent must select the strategy and method
• B4 — the complete B3 task is retained and factually correct but non-essential information is added to test distractor robustness

What You Can Inspect

Each public task page includes its identifier, domain, runtime requirements, input and output contracts, the tracked task_meta.yaml, and the exact source commit used to generate the page. Formal task directories in the public GitHub repository are metadata-only; versioned prompts and input data are distributed through the pinned benchmark datasets, while private reference answers and scoring configuration remain on the scoring side.

Publication Rules

Only release-ready tasks tracked in the canonical ASI-Bench repository appear in the public catalog. See Release Policy for the visibility and versioning rules.