Skip to content

Contribute a Task

Turn a meaningful scientific workflow from your field into an auditable ASI-Bench task. Build it step by step in the Web Guided Flow, or exact-sync an existing local Task with the CLI and return to the Web for final confirmation.
Web submission Start the browser Guided Flow ↗ New to the format? See how tasks work →

Is Your Task a Good Fit?

A strong contribution is:

  • scientifically meaningful — it represents a real computational workflow, not a toy puzzle
  • objectively scorable — expected artifacts can be checked numerically or structurally
  • reproducible — the same parameters and seed produce stable reference outputs
  • self-contained — runners receive the prompt and input data needed to attempt it
  • challenging for agents — the B1–B4 prompt ladder reveals a meaningful change in autonomy

Required local difficulty check before submission

While writing B1–B4

  • Keep every prompt model-agnostic. Do not name or tailor instructions to a specific Harness or model.
  • You do not need to choose a test configuration yet. Select it later in Local Testing, after the Task is ready.

Choose one approved configuration — use one of the following:

  1. Codex CLI · GPT-5.6 Sol · ultra
  2. Claude Code · Claude Opus 4.8 · max
  3. Claude Code · Fable 5 · max

Other Harness, model, or effort combinations may be useful during development, but they do not satisfy the submission requirement.

Run B1–B4 consistently

  • Use the same Harness, model, effort, and run settings for all four levels.
  • Complete one end-to-end run and record all four scores in Local Testing.

Score requirements

  • Every score must be a finite number from 0 to 100.
  • B1 and B2 have no independent score ceiling.
  • B3 and B4 must each be strictly below 40.
  • Scores must satisfy B1 ≥ B2 ≥ B3 ≥ B4. Equal adjacent scores are allowed.

Evidence to record

Record the selected configuration, four scores, stage timings, local environment, and scorer-replay evidence. Task submissions must run this contribution evidence in the OS sandbox (sandbox: os); other sandbox modes are not accepted for the Task submission gate. This evidence is author-supplied and is shown to reviewers as Not platform-verified.

Contribution Path

  1. Install the public toolkit with pip install asibench.
  2. Create a task scaffold, then add public metadata, B1–B4 prompts, deterministic generation, and scoring rules.
  3. Run the required pre-submit validation, then complete the same-configuration B1–B4 difficulty check with one approved maximum-effort option.
  4. Choose the Web Guided Flow or CLI Draft upload below, then complete final confirmation in the Portal.
  5. Respond to reviewer feedback until the task is accepted or closed.
# 1. Create a task scaffold
asibench task create --domain physics --name my_new_task

# 2. Validate the complete local task package
asibench validate --pre-submit tasks/physics/my_new_task/

# 3. Run one approved Harness/model/effort configuration across B1–B4
asibench difficulty-check --task physics.my_new_task

Choose How to Submit

Web submission

Use the browser Guided Flow when you want the Portal to guide you through metadata, prompts, evaluation, files, and local-testing evidence step by step. The final review screen shows every required field before submission.

CLI submission

Use this path when the complete Task directory already exists locally. Sign in once, validate the package, and upload it as an owner-only Draft:

asibench login
asibench validate --pre-submit tasks/<domain>/<your_task>/
asibench task submit --task-dir tasks/<domain>/<your_task>/

On first login, create a CLI token in Portal Settings and paste it into the hidden terminal prompt. The validated token is saved locally for later commands and is never passed as a command-line argument.

The submit command exact-syncs relative paths and file hashes, then opens the Draft page. Review the exact file list and imported fields in the Web Portal, and click Submit for Review to freeze the revision. CLI upload never bypasses this final Web confirmation.

Whichever path you choose, record the accepted Harness/model/effort configuration, valid B1–B4 scores, runtime, environment, and scorer-replay evidence before final submission. This evidence is author-supplied and remains labeled Not platform-verified during review.

Task submissions and result submissions are different

Use Contribute a Task when you are proposing a new scientific problem. If you have already run an agent on the public benchmark and want an official score, follow Submit Benchmark Results instead.

What Happens After Submission

The proposal remains private while maintainers check scientific value, reproducibility, task packaging, scoring reliability, and benchmark fit. Acceptance is not automatic: reviewers may request revisions before a task can enter a public release.

Contribution Recognition

Community contributors participate in building ASI-Bench rather than merely uploading external data. When a contributed task passes scientific and technical review and is incorporated into an official release, its contributor or contributors are included in that release's contributor list.

Next step Start a Web submission →