Scoring & Publication¶
The portal separates transport integrity, user confirmation, authoritative scoring, and public leaderboard publication.
Review Stages¶
- Upload — the portal safely extracts the archive and parses the schema v1 manifest.
- Integrity check — declared paths, byte sizes, and SHA-256 hashes are verified.
- Completeness check — missing outputs are summarized for each instance.
- Runner confirmation — the owner reviews the private draft and explicitly confirms it.
- Maintainer scoring — maintainers download the exact uploaded archive and run private authoritative scoring.
- Decision — a run can be scored, rejected with a reason, or held for clarification.
- Publication — only a scored run can be published to the public leaderboard.
Review Criteria¶
| Criterion | Requirement |
|---|---|
| Integrity | Manifest file hashes and byte sizes match the uploaded files |
| Completeness | Required outputs are present for every declared instance |
| Provenance | Agent, model, effort, sandbox, framework, and task identity are recorded |
| Comparability | Scores are compared only within compatible sandbox and run buckets |
| Score range | Authoritative mean score is recorded on the 0..100 scale |
| Privacy | No private reference answer or scoring configuration is accepted from the runner |
Statuses¶
| Status | Meaning |
|---|---|
uploaded |
Private draft awaiting runner confirmation |
received |
Confirmed and ready for maintainer processing |
scored |
Authoritative score recorded, not yet public |
published |
Visible on the public leaderboard |
rejected |
Not eligible; the owner can see the reason |
Reviews are handled on a best-effort basis. The submission detail page remains the source of truth for status and maintainer feedback.