Skip to content
Home / Catalog / Anonymous Deployment Prediction Set Audit

Anonymous Deployment Prediction Set Audit

  • Task ID: computer_science.deployment_prediction_sets
  • Domain: computer_science
  • Subdomain: machine_learning_reliability
  • Status: final
  • Benchmark set: seed42 (60 tasks)
  • Tags: uncertainty_quantification, prediction_sets, conformal_prediction, deployment_shift, subgroup_reliability, model_audit, calibration

Runtime and requirements

  • Estimated time: 60-150 minutes
  • Python: >=3.10
  • Packages: numpy>=1.24, pandas>=2.0, scipy>=1.11, matplotlib>=3.7
  • GPU required: no
  • Network required: no

Public input and output contract

Inputs

  • data/metadata.json (data): Agent-facing schema, anonymous feature columns, label columns, record counts, and output-file contract.
  • data/risk_spec.json (data): Risk target, label-specific miss costs, and prediction-set validity rules.
  • data/calibration_scores.csv (data): Black-box class probabilities for calibration records.
  • data/calibration_labels.csv (data): Observed labels for calibration records.
  • data/calibration_features.csv (data): Anonymous calibration features for audit and policy selection.
  • data/deployment_scores.csv (data): Black-box class probabilities for deployment records; deployment labels are hidden.
  • data/deployment_features.csv (data): Anonymous deployment features for audit and policy selection.

Outputs

  • analysis.py (code): End-to-end script that reads data/ and writes all required artifacts.
  • results/prediction_sets.csv (data): One row per deployment record with columns record_id,prediction_set. prediction_set is a space-separated list of label names.
  • results/policy_parameters.csv (data): Policy parameters used to construct the deployment sets.
  • results/audit_summary.csv (data): Risk estimates, set-size summaries, and policy audit metrics.
  • results/set_size_summary.csv (data): Per-deployment-record set sizes with columns record_id,set_size.
  • results/partition_diagnostics.csv (data): Partition-level audit diagnostics for the chosen policy.
  • results/distribution_diagnostics.csv (data): Distribution-comparison diagnostics used during policy selection.
  • results/calibration_diagnostics.csv (data): Binned calibration-score diagnostics used by the submitted policy.
  • results/constraint_audit.csv (data): Audited risk constraints and estimated violations.
  • results/set_size_tradeoff.png (figure): Figure summarizing risk-size tradeoffs.
  • results/partition_audit.png (figure): Figure summarizing group-level risk.
  • results/score_diagnostics.png (figure): Figure summarizing score or feature diagnostics.

Public repository files

View this task in ASI-Bench

Formal benchmark task directories in the public repository are metadata-only. Versioned prompts and inputs are distributed through the pinned benchmark dataset.

Provenance

This task is included in the current seed42 benchmark set at dataset revision f11a199f71fb4b854ac43a1bf548d5df519141a9.

This page is generated only from files tracked in the public ASI-Bench repository at commit f18382f03faf. Submission bundles, run logs, private scoring configuration, and private reference answers are not read by this page generator.