Contribute a Task¶
Turn a meaningful scientific workflow from your field into an auditable ASI-Bench task. You can author and validate it locally, then send it through the submission portal for maintainer review.
Is Your Task a Good Fit?¶
A strong contribution is:
- scientifically meaningful — it represents a real computational workflow, not a toy puzzle
- objectively scorable — expected artifacts can be checked numerically or structurally
- reproducible — the same parameters and seed produce stable reference outputs
- self-contained — runners receive the prompt and input data needed to attempt it
- challenging for agents — the B1–B4 prompt ladder reveals a meaningful change in autonomy
Contribution Path¶
- Install the public toolkit with
pip install asibench. - Create a task scaffold, then add public metadata, B1–B4 prompts, deterministic generation, and scoring rules.
- Run the required pre-submit validation and the same-model B1–B4 difficulty check.
- Open the task-submission form from the CLI and complete the private proposal in the Portal.
- Respond to reviewer feedback until the task is accepted or closed.
# 1. Create a task scaffold
asibench task create --domain physics --name my_new_task
# 2. Validate the complete local task package
asibench validate --pre-submit tasks/physics/my_new_task/
# 3. Run the required same-model trial across B1, B2, B3, and B4
asibench difficulty-check --task physics.my_new_task
# 4. Open the Portal task-submission form
asibench task submit
asibench task submit opens the guided Portal form; it does not upload task files or create a draft from the terminal. Upload the task files and record the B1–B4 local-test evidence in the browser.
Task submissions and result submissions are different
Use Contribute a Task when you are proposing a new scientific problem. If you have already run an agent on the public benchmark and want an official score, follow Submit Benchmark Results instead.
What Happens After Submission¶
The proposal remains private while maintainers check scientific value, reproducibility, task packaging, scoring reliability, and benchmark fit. Acceptance is not automatic: reviewers may request revisions before a task can enter a public release.