Skip to main content
Use benchmark commands to compare agents on published task sets. A benchmark config selects tasks, agent harnesses, models, and execution settings; it does not reference your workspace rollout code.
Benchmarks are in beta and enabled per account; until yours is enabled the CLI returns Benchmarks is not available for your account. The benchmark must also already be available in the workspace. Osmosis-managed benchmarks are added automatically; add a Harbor registry benchmark from the Platform before using it in a CLI config.

​
Quick Start

Run these commands inside the workspace repository:
Then inspect the run:

​
Benchmark Config

Start with one task, one agent, and one attempt. See Config Files for task sets, categories, multiple agents, custom endpoints, hosted models, judges, verifiers, and execution controls.
configs/benchmark/terminal-bench-smoke.toml
api_key_secret is a Platform secret record name, not a credential value. Register every model, harness, judge, and verifier secret referenced by the config before submission.
Benchmark runs incur sandbox and model charges, and model spend uses your provider keys. Start with one task and one attempt before increasing the selection or concurrency.

​
How It Works

1

Inspect the benchmark

benchmark list shows what the workspace can run. benchmark info reports the exact key, task manifest, categories, named task sets, supported harnesses, judge requirements, pass threshold, leaderboard, and existing runs.
2

Select tasks and agents

Omit [tasks] to run the full benchmark, or select explicit task names, categories, or a published task set. Add one [[agents]] entry per agent you want to compare.
3

Preview and submit

benchmark submit validates the benchmark, task selection, harnesses, models, execution controls, and secret references, then previews the resolved run before confirmation.
4

Execute and score

Osmosis runs each agent against the same task selection and attempt budget in benchmark-managed environments. The run records outcomes, pass@k metrics, cost, duration, tokens, logs, and artifacts when available.

​
Commands

See the Command Reference for the full flag list.

​
Download Run Outputs

The default download contains summary and task-result CSV files. Optional selectors add artifacts and logs. Downloads use a fixed run directory, skip complete local files unless --overwrite is set, and can be resumed by running the same command again.

​
Next Steps

Config Files

Configure tasks, agents, models, judges, and execution.

Platform Benchmarks

Add benchmarks, inspect leaderboards, and understand result eligibility.

Command Reference

Review every benchmark command and option.
Last modified on August 10, 2026