Skip to main content
Benchmark runs compare one or more agent harness and model combinations on a benchmark managed by the Platform. The CLI submits an Osmosis TOML config; the Platform owns the benchmark source, task environment, execution, and result collection.
Add the benchmark to the workspace in the Platform before submitting it from the CLI. [experiment].benchmark resolves a benchmark already in the workspace by key, name, or ID, not a local dataset path.

​
Inspect the benchmark

List the benchmarks available in the current workspace, then inspect the one you plan to run:
benchmark info shows the key and exact name, a source_url linking to the upstream project (or the adapter repository when the benchmark is adapted), task and category counts, named task sets, harness and judge requirements, pass threshold, and any implicit required secret record names, followed by the benchmark’s leaderboard and the workspace’s runs on it. When a benchmark’s published scores were measured on a specific harness, that harness is reported as the default. Terminal-Bench 2.1’s is terminus-2, and a run on another harness is not comparable with those scores. Use JSON output to inspect the complete task manifest and required_secret_names before choosing task_names or categories:
Every task in the JSON manifest includes difficulty, whose value is easy, medium, hard, or null. A null value means the benchmark source did not provide a difficulty; do not infer one. For HLE, the command marks parity as the recommended named task set. Omit [tasks] to select the full benchmark. A Harbor registry benchmark’s task list pages in from the registry after the benchmark is added. While that is in progress the Last Run column reports the sync and its task progress, and submitting against it fails until it completes. If the sync failed, benchmark list reports the reason there and unavailable under Tasks, with a sync_error in JSON output; retry it from the benchmark’s page in the Platform. The reported platform_url opens that page, with its leaderboard and runs.

​
Create a config

Copy the workspace template and edit it:
For the first run, select one task and one agent:
configs/benchmark/terminal-bench-smoke.toml
See Configuration Files for provider, endpoint, hosted model, task-filter, and execution fields.

​
Register credentials

api_key_secret, harness_api_key_secret, and judge_api_key_secret contain Platform secret record names, never credential values. Create every record referenced by the config before submitting:
Also create every record named in the benchmark info response’s required_secret_names:
These are implicit benchmark requirements and are not repeated in the TOML. The response returns record names only. For example, HLE includes HF_TOKEN. Personal scope is the default. Use --scope workspace for a credential shared with workspace members who can submit runs. Cursor CLI and Mini SWE-agent authenticate separately from the model. Set that agent’s harness_api_key_secret to CURSOR_API_KEY for Cursor CLI, or to MSWEA_API_KEY for Mini SWE-agent. Those are the variables the harnesses read, and any other value is rejected at submit.
Configure only the record needed by the selected harness. Harnesses other than cursor-cli and mini-swe-agent reject harness_api_key_secret. See Configuration Files for a complete agent example. HLE and GDPVal use an LLM judge and require judge_api_key_secret. Set it to a Platform secret record name; judge_model is optional and uses the benchmark default when omitted. For HLE, create the judge record and the HF_TOKEN record reported by required_secret_names:
Then add the judge record name to the HLE config’s existing [execution] table:
HF_TOKEN is reserved by the benchmark runner for every benchmark, not only HLE. Never put it in top-level [env] or any [agents.env], and never use it as a model’s api_key_secret; for HLE, store the dataset credential only in the HF_TOKEN Platform secret record. A submit with a missing secret fails with the exact record names to create.
Before submitting HLE, add task_set = "parity" under [tasks]. We recommend the published parity set for HLE runs; omitting [tasks] selects the full HLE benchmark.

​
Submit the run

The confirmation preview shows the benchmark, task selection, agent models, attempts, concurrency, and secret scopes. To confirm non-interactively after reviewing the config:
The response includes:
  • Generated run id and name
  • Initial status
  • Resolved task_count
  • Submission timestamp in created_at
  • platform_url for the benchmark run in the Platform

​
Manage the run

Run lifecycle commands live under benchmark runs. List benchmark runs in the current workspace, which shows the same columns as the Platform’s runs table, including each run’s agent count and best pass@1, then inspect the submitted run by name or ID:
runs info reports the run’s status, progress, duration, and result totals, and scores every agent the way the run page does: rank, pass@1 with its confidence interval, the deepest pass@k, and per-task cost, time, and tokens. Use runs logs to inspect lifecycle events and diagnose failures. Logs are returned oldest first within each page; pass the JSON response’s next_cursor to page further back:
Stop a pending, queued, or running benchmark after reviewing the confirmation:
Download the run summary and task-level results, or select additional output types:
Downloads use a run-scoped directory under .osmosis/benchmarks/ by default and resume by skipping complete files. Available types are summary, results, artifacts, logs, and all. The layout is fixed:
Outputs are unavailable while a run is pending or queued. A running run downloads a current snapshot; pass --overwrite when refreshing files that may have changed without changing size. You can also open the platform_url returned by submit or runs info to follow progress, compare agents, and inspect task-level results in the Platform.

​
Expand the run

After the smoke run starts and produces expected results:
  1. Add explicit task_names, select categories, use a published parity task set, or omit [tasks] for the full benchmark.
  2. Increase max_concurrent_attempts within your workspace limits.
  3. Add another [[agents]] table to compare harness or model combinations.
  4. Review the new task and attempt counts before confirming the paid run.
Benchmark runs can incur model and sandbox charges. Prefer a one-task smoke run before submitting a full benchmark, and do not pass --yes until the run scope has been reviewed.