Skip to content

Run Your First Task

This walkthrough uses public task instances and produce-only mode—the same path an external benchmark runner uses before submitting results for official scoring.

Prerequisites

  • Install ASI-Bench
  • prepare an agent command that reads a task workspace and writes the requested outputs
  • use Linux for the isolated linux_ns custom-command path, or use none only for trusted local testing

Step 1: Download Task Instances

asibench task pull --output-dir hf_instances/

The downloaded directory contains public prompts and input data, not private scoring material.

Step 2: Run Your Agent

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

Replace the example --agent-cmd with your own non-interactive command. The {workspace} placeholder is replaced with the prepared task directory for each run.

Option Purpose
--no-score Produce artifacts without access to private reference answers
--instances-dir Read downloaded prompts and input data from this directory
--agent-cmd Launch your agent against each prepared workspace
--sandbox linux_ns Isolate a custom file-exchange agent with Linux namespaces
--output-dir Store outputs and run provenance for later submission

Step 3: Check the Output Directory

Confirm that out/ contains run metadata and the outputs requested by the task instances. Missing declared files will be reported when the bundle is uploaded.

Step 4: Submit the Run

asibench submit --results-dir out/

Open the returned link and confirm the draft after reviewing its integrity and completeness summary.

Next Steps