Skip to content
Home / Catalog / Bootstrap Confidence-Interval Engine Forensics

Bootstrap Confidence-Interval Engine Forensics

  • Task ID: computer_science.bci_forensics
  • Domain: computer_science
  • Subdomain: statistical_computing
  • Status: final
  • Benchmark set: seed42 (60 tasks)
  • Tags: bootstrap, confidence_intervals, statistical_computing, uncertainty_quantification, simulation_diagnostics, forensic_audit

Runtime and requirements

  • Estimated time: 90-180 minutes
  • Python: >=3.10
  • Packages: numpy>=1.24, matplotlib>=3.7
  • GPU required: no
  • Network required: no

Public input and output contract

Inputs

  • data/study_manifest.json (data): Agent-facing study metadata, confidence level, case ids, statistic definitions, regime tags, public metric columns, and output schema notes.
  • data/observed_samples.csv (data): Observed samples with columns case_id, observation_id, value.
  • data/resample_panels.npz (data): Deterministic observed-case resampling index panels keyed by case id.
  • data/calibration_samples.npz (data): Monte Carlo calibration samples keyed by case id.
  • data/calibration_resample_panels.npz (data): Deterministic calibration resampling index panels keyed by case id.
  • data/engine_interval_panels.npz (data): Anonymous candidate engine interval endpoints for observed cases, calibration replications, and neutral auxiliary endpoint panels.
  • data/probe_manifest.json (data): Public probe family names, array shapes, and diagnostic column names.

Outputs

  • analysis.py (code): End-to-end analysis script that reads data/ and writes all requested outputs under results/.
  • results/engine_audit.csv (data): One row per anonymous engine with coverage, tail, width, neutral probe metrics, and aggregate engine-score diagnostics.
  • results/failure_modes.json (data): Selected engine id and a failure-mode label for every anonymous engine.
  • results/selected_engine.json (data): Selected reliable engine id, selected label, and selected engine score.
  • results/interval_summary.csv (data): Observed-case endpoints from the selected engine plus baseline endpoints and width ratios.
  • results/coverage_audit.csv (data): Per case and engine empirical coverage, mean width, tail miss rates, and coverage score.
  • results/probe_audit.csv (data): Per case, probe family, and engine diagnostic values and probe scores.
  • results/case_probe_profile.csv (data): Per case and engine probe-signal profile used to show cross-family diagnostic reasoning.
  • results/failure_evidence.csv (data): One row per engine and public failure-mode label with evidence score, label rank, and predicted-label flag.
  • results/family_ablation.csv (data): Probe-family ablation audit showing how rankings change when one evidence family is removed.
  • results/reliability_report.json (data): Structured report with selected engine, aggregate diagnostics, ranking, and per-case warnings.
  • results/engine_overview.png (figure): Diagnostic figure summarizing anonymous engine reliability; low scoring weight.

Public repository files

View this task in ASI-Bench

Formal benchmark task directories in the public repository are metadata-only. Versioned prompts and inputs are distributed through the pinned benchmark dataset.

Provenance

This task is included in the current seed42 benchmark set at dataset revision f11a199f71fb4b854ac43a1bf548d5df519141a9.

This page is generated only from files tracked in the public ASI-Bench repository at commit f18382f03faf. Submission bundles, run logs, private scoring configuration, and private reference answers are not read by this page generator.