Release Notes¶
Updates¶
2026-08-16 — GLM-5.3 Result Added¶
The supplied aggregate leaderboard now contains 18 ranked harness–model combinations. Claude Code with GLM-5.3 at Max effort was added with B1 63.05, B2 35.45, B3 33.09, B4 35.01, and an overall score of 41.65. The model release metadata links to Z.ai's official August 14, 2026 launch announcement.
2026-08-11 — Public Repository Rebuilt with Metadata-Only Task Directories¶
The canonical ASI-Bench repository was rebuilt with clean Git history. The website catalog is now pinned to public commit c9c1ba9, and every generated task page links only to the task_meta.yaml that is present in the rebuilt repository.
Formal task directories on GitHub are metadata-only. Versioned prompts and benchmark inputs continue to come from the pinned Apexintelligence-AI/ASI-Bench-seed42 dataset; private generators, scorers, references, and submission data are not copied into the public repository or website.
2026-08-04 — 60-Task Benchmark Catalog Published¶
The Task Catalog and all generated detail pages are now aligned to the 60-task Apexintelligence-AI/ASI-Bench-seed42 benchmark set. Membership is pinned to dataset revision f11a199f71fb4b854ac43a1bf548d5df519141a9, while public metadata and file links remain pinned to validated ASI-Bench commit 52f8115.
Six restored benchmark tasks were added and four public task definitions outside the current benchmark set were removed. Example tasks remain excluded. Every published detail page is generated from public task_meta.yaml metadata. Versioned B1–B4 prompts and inputs come from the pinned benchmark dataset; private scoring configuration, reference answers, ground-truth generators, and run data are not read or copied.
2026-08-03 — Aggregate Leaderboard Refreshed¶
The leaderboard now uses the latest supplied all-combinations B1–B4 aggregate snapshot. It contains 17 ranked harness–model combinations: 16 with three adopted source rounds and Claude Opus 5 with R1 only.
Claude Fable 5 no longer appears in the supplied ranking. Claude Code + DeepSeek V4 Flash, OpenHands + DeepSeek V4 Flash, and Kimi Code + Kimi K3 were added, and every retained row received refreshed B1–B4 and overall values. The evaluation task count is confirmed as 60; per-result score-cell coverage remains unavailable and is not inferred.
2026-08-03 — Public Task Catalog Updated¶
The public catalog is now pinned to validated ASI-Bench commit 52f8115. It contains 58 release-ready tasks across 11 domains, generated only from tasks whose public metadata status is final.
Nine newly final tasks were added across biostatistics, computer science, earth science, mathematics, physics, and robotics. medicine.jama_id0014_malignant_an was removed from the public catalog after its upstream status changed from final to test. Abandoned and test-only examples remain excluded. Leaderboard scores continue to use their separately versioned evaluation task set.
2026-07-29 — Completed 61-Task Score Snapshot Published¶
The leaderboard now uses the supplied 61-task aggregate score snapshot. Fifteen agent–model combinations have complete B1–B4 coverage for every adopted round listed in the source and are ranked publicly.
Most combinations aggregate three adopted rounds (732 score cells). Claude Opus 5 and Claude Fable 5 currently use R1 (244 score cells) and are labeled accordingly. Kimi Code + Kimi K3 is omitted because it covers only 48 of 61 tasks (384/732 cells, 52.46%). Evaluation date and benchmark, task-set, scorer, runner, exact model, and harness version identifiers were not included in the source and remain visibly marked as missing.
2026-07-28 — Completed 48-Task Score Snapshot Published¶
The leaderboard now uses the supplied 48-task aggregate score snapshot. Fourteen agent–model combinations contain the complete 48 tasks × B1–B4 × 3 rounds matrix (576 score cells each) and are ranked publicly.
Two additional combinations contain only one round. They are intentionally omitted from the public results and ranking so every displayed row has comparable three-round coverage. Evaluation date and benchmark, task-set, scorer, runner, exact model, and harness version identifiers were not included in the source and remain visibly marked as missing.
2026-07-26 — Public Release Snapshot Pinned¶
The website task catalog was pinned to validated public ASI-Bench commit 74c846d. That source snapshot contained 56 final tasks across 11 domains before the website release denylist was applied.
At that release, leaderboard scores were sourced separately from a supplied 57-task evaluation snapshot. Later score snapshots remain separate from repository task metadata; the latest is recorded above. A newer repository commit is published only after it passes the final-task and Example allowlist checks.
2026-07-14 — Public Task Source Pinned to ASI-Bench¶
The website task catalog now uses a checked-in allowlist synchronized from the public ASI-Bench repository. At that release, task pages identified the source commit and linked to the files tracked there. The metadata-only repository policy introduced on 2026-08-11 supersedes the earlier file layout.
Tasks that exist only in an internal working repository are no longer eligible for automatic publication. Public website statistics also exclude internal task inventory counts.
2026-04-22 — Public Website Scaffold Established¶
The benchmark now has a real MkDocs-based web/ scaffold with:
- homepage
- overview
- task catalog
- leaderboard section
- getting-started pages
- FAQ
- paper and news pages
This moved the website from planning documents into an actual buildable public site structure.
2026-04-22 — Public Task Catalog Limited to test Tasks¶
The generated public task catalog was tightened so the website no longer mirrors the full internal task tree. The public website now shows only the test subset.
Current public task and domain counts are generated from the latest repo metadata rather than maintained by hand.
2026-04-22 — Public Task Summaries Sanitized¶
The task catalog exporter now derives safer public summaries automatically and avoids leaking template-heavy prompt text such as unresolved {{ ... }} placeholders.
This makes the public task cards and catalog entries read more like benchmark summaries and less like raw prompt exports.
2026-04-22 — Homepage, Catalog, and Leaderboard Became Data-Driven¶
The site now renders generated JSON directly for:
- homepage stats
- featured public tasks
- task catalog filtering/search
- leaderboard rendering and verified review snapshot handling
This means the public site is now tied to generated benchmark metadata instead of relying entirely on hardcoded page text.