Quick Start

Run ASI-Bench with your own agent, collect its output artifacts, and submit the run for official scoring. Public benchmark runners do not receive reference answers or score locally. When the run is for public `seed42` Results, the effective sandbox must be `os`.

Before You Start

You need Python 3.11+ and an agent that can read a prepared workspace and write the requested output files. Custom file-exchange commands use linux_ns on Linux or none for local testing only. For an official seed42 Results submission, use a compatible built-in adapter with Docker-based os isolation.

1. Install the Runner

pip install asibench
asibench --help

2. Pull Public Task Instances

asibench task pull --output-dir hf_instances/

This downloads public prompts and input data. Private reference answers and scoring configuration are not included.

3. Run Your Agent

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

The framework prepares each task workspace, runs your command, collects declared outputs, and records provenance for the submission bundle. All four prompt levels are evaluated unless you explicitly pass --prompt-levels. This custom-command example is for local testing; for public seed42 Results, use a compatible built-in adapter with --sandbox os because the Portal rejects other effective sandbox modes.

4. Submit for Official Scoring

asibench submit --results-dir out/

With an upload endpoint configured, the command prints a confirmation link. Open it, check the parsed run summary, and click Confirm. Uploading alone leaves the run as a private draft. Without an endpoint, it creates a local archive and reports NOT SUBMITTED — nothing was uploaded.

Connect an agent Run with your agent → Finished a run? Submit benchmark results →

Contributing a task is a separate workflow

If you want to add a new scientific problem to ASI-Bench, start with Contribute a Task. Do not use the result-submission workflow.