Skip to content

Submit Benchmark Results

Use the ASI-Bench CLI to package and upload a produce-only run for integrity checks, confirmation, maintainer scoring, and possible publication on the official leaderboard.

Submitting a new Task instead?

Task contribution is a separate workflow. Choose the Web Guided Flow or upload a local Draft with the CLI: start on the Web, or see the Task contribution guide for the CLI commands. Both paths return to the Portal to review files and imported fields before Submit for Review.

Current Workflow

  1. Produce public benchmark outputs with asibench run --no-score.
  2. Run the canonical submit command. It builds a schema-v1 .tar.gz bundle and, when a Portal endpoint is configured, uploads it as a private draft.
  3. Open the confirmation link printed by the CLI, review the integrity and completeness report, then click Confirm.
  4. Maintainers score the confirmed bundle against the private reference set.
  5. Approved scores may be published to the official leaderboard.
asibench submit --results-dir out/

Do not manually construct, rename, or repackage a raw results folder, and do not upload a ZIP file. The CLI records the current schema, checksums, output bytes, instance identity, and run provenance required for official evaluation.

When no upload endpoint is configured

The CLI still creates a local bundle, but prints NOT SUBMITTED — nothing was uploaded. Configure ASIBENCH_SUBMIT_ENDPOINT or pass --endpoint <url>, then run the command again to upload.

Uploading is not the last step

A successful upload creates a private draft. Open the confirmation link returned by the CLI and click Confirm before the run enters the maintainer scoring queue.

Required Properties

Requirement Details
Archive .tar.gz with regular files only; no links, devices, or parent paths
Schema submission_schema_version must be 1
Integrity Every included file is indexed with its relative path, byte size, and SHA-256
Instances At least one instance with identity, prompt level, provenance, and outputs
Sandbox Effective sandbox mode is recorded in instance provenance; public ASI-Bench-seed42 Results require os and other modes are rejected
Privacy Reference answers and private scoring configuration are not included