Skip to content

Run Your First Task

This walkthrough uses a custom command with public task instances in produce-only mode for local smoke testing. It is not the official `seed42` submission path: official Results require a compatible built-in adapter and `--sandbox os`.

Prerequisites

  • Install ASI-Bench
  • prepare an agent command that reads a task workspace and writes the requested outputs
  • use Linux for the isolated linux_ns custom-command path, or use none only for trusted local testing; neither mode is accepted for official seed42 Results

Step 1: Download Task Instances

asibench task pull --output-dir hf_instances/

The downloaded directory contains public prompts and input data, not private scoring material.

Step 2: Run Your Agent

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'python my_agent.py --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

Replace the example --agent-cmd with your own non-interactive command. The {workspace} placeholder is replaced with the prepared task directory for each run.

Option Purpose
--no-score Produce artifacts without access to private reference answers
--instances-dir Read downloaded prompts and input data from this directory
--agent-cmd Launch your agent against each prepared workspace
--sandbox linux_ns Isolate a custom file-exchange agent with Linux namespaces
--output-dir Store outputs and run provenance for later submission

Step 3: Check the Output Directory

Confirm that out/ contains run metadata and the outputs requested by the task instances. Missing declared files will be reported when the bundle is uploaded.

Step 4: Submit the Run

asibench submit --results-dir out/

This custom-command output is suitable for local inspection. To submit official seed42 Results, rerun with a compatible built-in adapter and --sandbox os, then open the returned link and confirm the draft after reviewing its integrity and completeness summary.

Next Steps