Skip to content

Understand Results

An external ASI-Bench run produces artifacts and provenance, not official local scores. Maintainers compute the published scores with private references after submission.

What Your Local Run Contains

The directory passed to --output-dir contains the run-level metadata, per-instance records, requested task artifacts, and agent execution logs needed to package and review the run. Exact filenames can evolve with the runner schema, so treat the CLI-generated directory as one submission unit rather than hand-editing individual records.

Before submitting, check that:

  • each intended task and prompt level has a result record;
  • the task-requested output files exist;
  • agent, model, effort, sandbox, framework version, and timeout are recorded correctly;
  • the run completed without an unexpected environment or adapter failure.

Produce-Only Is Not a Score

External runners use:

asibench run --no-score ...

This prevents access to private reference answers and scoring configuration. The local directory can confirm execution and artifact completeness, but it cannot provide the authoritative leaderboard score.

Task authors are a separate case: they can use private task files with --score while validating a task they own. Those private author scores are not external leaderboard submissions.

Review a Result Directory

asibench report --results-dir out/

Use the report to inspect the generated run records and spot missing or failed instances before packaging. Do not interpret a produce-only report as an official benchmark ranking.

Submit for Official Scoring

asibench submit --results-dir out/

When an upload endpoint is configured, the command creates a private draft and returns a review link. Verify the parsed completeness and provenance summary, then confirm the draft to enter the maintainer scoring queue. Uploading without confirmation does not publish or score the run.

Without an endpoint, the command builds a local archive and prints NOT SUBMITTED — nothing was uploaded together with its path.

What the Website Publishes

The public leaderboard should publish only reviewed, fully completed evaluations. Each displayed result needs:

  • the agent or harness and model identity;
  • an overall score and B1–B4 means;
  • complete task × prompt-level × round coverage;
  • a benchmark/task-set version and scoring-policy version;
  • enough provenance to establish that compared runs used compatible policies.

Partial, pending, and in-progress runs belong in private operational tooling, not the public overview.

Next Steps