FAQ¶
What does project-level mean?¶
ASI-Bench evaluates Agent×Model systems on long-horizon, project-level scientific research rather than isolated questions or coding prompts. A system must understand the objective, inspect data, choose a method, implement and refine a solution, and produce artifacts that can be checked objectively.
Why are there B1-B4 prompt levels?¶
The prompt ladder measures how far an agent can proceed as human methodological guidance is withdrawn. B1 provides the full method and procedure; B2 retains the intended method and constraints but leaves implementation open; B3 provides only the objective, inputs, constraints, and required outputs; and B4 keeps B3 unchanged while adding factually correct but non-essential context.
Why does the benchmark emphasize structured outputs?¶
Structured outputs make evaluation more reproducible and easier to compare across agents and runs.
What determines whether a task appears on the website?¶
The exact frozen Revision must be approved and merged into the private benchmark through an administrator-reviewed Draft PR. Then only its public-safe files may be published to public ASI-Bench main with status: final and included in a manually refreshed website snapshot. Approval or repository presence alone never publishes it automatically.
Are all tasks on the website fully released benchmark tasks?¶
Every catalog entry is release-ready public benchmark material at the source commit shown on its detail page. The catalog can still grow as new tasks pass review and enter the public repository.
How are results stored?¶
The framework stores structured result artifacts such as:
run_metadata.json- per-instance result JSON files
- aggregated
batch_recordsoutputs such asbatch_overview.json,task_scoreboard.json, andtask_level_long.json
These artifacts are what make leaderboard rendering and result inspection reproducible.
Why is the public leaderboard not final yet?¶
The website can render leaderboard data, and the current page may show a conservative repo-verified review snapshot derived from checked repo records. That snapshot is useful for website review and demonstration, but a final public official baseline snapshot has not been published yet.
What is the difference between official and community results?¶
The distinction is:
- official leaderboard: reviewed and accepted benchmark runs
- submitted runs: private drafts and uploaded bundles awaiting maintainer review in the task submission portal
Only reviewed and explicitly published aggregate scores enter the public leaderboard.
Can I submit tasks or benchmark results?¶
Yes. Use the task submission portal to propose a new scientific task or upload benchmark results for review. Draft proposals and uploaded result bundles remain private; only reviewed and explicitly published scores enter the public leaderboard.
Why are some task detail pages still sparse?¶
Task pages are generated from the canonical public task_meta.yaml files and show the runtime and input/output contract. Formal task directories on GitHub are metadata-only after the clean-repository migration; versioned prompts and input data come from the pinned Hugging Face benchmark datasets.