Skip to content

Run with Your Agent

Connect an agent through a simple workspace command or a built-in adapter. External benchmark runs stay in produce-only mode and are scored after submission.

Choose an Integration Path

Custom workspace command

Use --agent-cmd when your agent can read a prepared workspace and write the requested files back into it:

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent-cmd 'my-agent --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

{workspace} is replaced with the task workspace path. Custom file-exchange commands support linux_ns for isolated Linux runs and none for trusted local testing.

Built-in adapter

Use --agent and --agent-config when the framework already has a compatible adapter:

asibench run --no-score \
  --instances-dir hf_instances/ \
  --agent kimi_code_cli \
  --agent-config '{"model":"kimi-k2.7"}' \
  --sandbox os \
  --output-dir out/

Docker-based os isolation requires an adapter that supports it; it is not the custom --agent-cmd path.

Built-in CLI Agents

Agent Installation Authentication
Claude Code npm install -g @anthropic-ai/claude-code Anthropic API key or claude /login
Codex CLI npm install -g @openai/codex OpenAI API key
Kimi Code npm install -g @moonshot-ai/kimi-code Moonshot API key or kimi /login
CodeWhale Follow the CodeWhale project documentation DeepSeek API key by default
AntiGravity npm install -g @anthropic-ai/antigravity Experimental adapter

For Kimi Code, use the npm package shown above. The unrelated kimi-cli PyPI package has incompatible arguments.

Sandbox Compatibility

Mode Use it with Notes
linux_ns Custom --agent-cmd Recommended isolated custom-command path on Linux
none Custom --agent-cmd Trusted local testing only; no isolation
task / os Compatible built-in adapters Adapter-managed environment or Docker isolation

The effective sandbox, agent, model, framework version, and other run settings are recorded as provenance.

Run Defaults That Matter

  • asibench run evaluates B1, B2, B3, and B4 by default. Pass --prompt-levels only when you intentionally want a subset.
  • The execution timeout comes from asibench run --timeout and defaults to 10,800 seconds.
  • Public runners use --no-score; private reference answers and scorer configuration are not distributed.
  • Retired orchestration commands such as batch-run, eval, and pipeline are not part of the public CLI.

To smoke-test a smaller public subset before a full run, select published task IDs explicitly, for example:

asibench run --no-score \
  --instances-dir hf_instances/ \
  --tasks physics.lid_driven_cavity_flow,physics.vpfm_leapfrog \
  --agent-cmd 'my-agent --workspace {workspace}' \
  --sandbox linux_ns \
  --output-dir out/

Image Input

Framework-managed model proxies default to text-only input. If the connected endpoint accepts visual input, opt in with "supports_image_input": true in --agent-config. Direct endpoints outside a framework proxy remain the connected agent's responsibility.

Submit the Outputs

asibench submit --results-dir out/

With an endpoint configured, review and confirm the returned draft before it enters the scoring queue. Without an endpoint, the command only creates a local bundle and explicitly reports that nothing was uploaded.

Next Steps