Quick Start¶

Run ASI-Bench with your own agent, collect its output artifacts, and submit the run for official scoring. Public benchmark runners do not receive reference answers or score locally.

Before You Start¶

You need Python 3.11+ and an agent that can read a prepared workspace and write the requested output files. Custom file-exchange commands use linux_ns on Linux or none for trusted local testing; Docker-based os isolation requires a compatible built-in adapter.

1. Install the Runner¶

pip install asibench
asibench --help

2. Pull Public Task Instances¶

asibench task pull --output-dir hf_instances/

This downloads public prompts and input data. Private reference answers and scoring configuration are not included.

3. Run Your Agent¶

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

The framework prepares each task workspace, runs your command, collects declared outputs, and records provenance for the submission bundle. All four prompt levels are evaluated unless you explicitly pass --prompt-levels.

4. Submit for Official Scoring¶

asibench submit --results-dir out/

With an upload endpoint configured, the command prints a confirmation link. Open it, check the parsed run summary, and click Confirm. Uploading alone leaves the run as a private draft. Without an endpoint, it creates a local archive and reports NOT SUBMITTED — nothing was uploaded.

Connect an agent Run with your agent → Finished a run? Submit benchmark results →

Contributing a task is a separate workflow

If you want to add a new scientific problem to ASI-Bench, start with Contribute a Task. Do not use the result-submission workflow.