> ## Documentation Index
> Fetch the complete documentation index at: https://docs.osmosis.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Runs

> Configure, submit, monitor, and download managed benchmark runs from the Osmosis CLI

Benchmark runs compare one or more agent harness and model combinations on a benchmark managed by the Platform. The CLI submits an Osmosis TOML config; the Platform owns the benchmark source, task environment, execution, and result collection.

<Note>
  Add the benchmark to the workspace in the Platform before submitting it from the CLI. `[experiment].benchmark` resolves a benchmark already in the workspace by key, name, or ID, not a local dataset path.
</Note>

## Inspect the benchmark

List the benchmarks available in the current workspace, then inspect the one you plan to run:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark list
osmosis benchmark info terminal-bench-2-1
```

`benchmark info` shows the key and exact name, a `source_url` linking to the upstream project (or the adapter repository when the benchmark is adapted), task and category counts, named task sets, harness and judge requirements, pass threshold, and any implicit required secret record names, followed by the benchmark's leaderboard and the workspace's runs on it. When a benchmark's published scores were measured on a specific harness, that harness is reported as the default. Terminal-Bench 2.1's is `terminus-2`, and a run on another harness is not comparable with those scores. Use JSON output to inspect the complete task manifest and `required_secret_names` before choosing `task_names` or `categories`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json benchmark info terminal-bench-2-1
```

Every task in the JSON manifest includes `difficulty`, whose value is `easy`, `medium`, `hard`, or `null`. A `null` value means the benchmark source did not provide a difficulty; do not infer one.

For HLE, the command marks `parity` as the recommended named task set. Omit `[tasks]` to select the full benchmark.

A Harbor registry benchmark's task list pages in from the registry after the benchmark is added. While that is in progress the Last Run column reports the sync and its task progress, and submitting against it fails until it completes. If the sync failed, `benchmark list` reports the reason there and `unavailable` under Tasks, with a `sync_error` in JSON output; retry it from the benchmark's page in the Platform. The reported `platform_url` opens that page, with its leaderboard and runs.

## Create a config

Copy the workspace template and edit it:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
cp configs/benchmark/default.toml configs/benchmark/terminal-bench-smoke.toml
```

For the first run, select one task and one agent:

```toml configs/benchmark/terminal-bench-smoke.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
benchmark = "terminal-bench-2-1"

[tasks]
task_names = ["terminal-bench/git-multibranch"]

[[agents]]
harness = "codex"

[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"

[execution]
attempts_per_task = 1
max_concurrent_attempts = 1
timeout_multiplier = 1.0
max_retries = 0
```

See [Configuration Files](/cli/config-files#benchmark-config) for provider, endpoint, hosted model, task-filter, and execution fields.

## Register credentials

`api_key_secret`, `harness_api_key_secret`, and `judge_api_key_secret` contain Platform secret record names, never credential values. Create every record referenced by the config before submitting:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set OPENAI_API_KEY
```

Also create every record named in the `benchmark info` response's `required_secret_names`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set NAME
```

These are implicit benchmark requirements and are not repeated in the TOML. The response returns record names only. For example, HLE includes `HF_TOKEN`.

Personal scope is the default. Use `--scope workspace` for a credential shared with workspace members who can submit runs.

Cursor CLI and Mini SWE-agent authenticate separately from the model. Set that agent's `harness_api_key_secret` to `CURSOR_API_KEY` for Cursor CLI, or to `MSWEA_API_KEY` for Mini SWE-agent. Those are the variables the harnesses read, and any other value is rejected at submit.

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set CURSOR_API_KEY
osmosis secret set MSWEA_API_KEY
```

Configure only the record needed by the selected harness. Harnesses other than `cursor-cli` and `mini-swe-agent` reject `harness_api_key_secret`. See [Configuration Files](/cli/config-files#agents-and-agentsmodel) for a complete agent example.

HLE and GDPVal use an LLM judge and require `judge_api_key_secret`. Set it to a Platform secret record name; `judge_model` is optional and uses the benchmark default when omitted. For HLE, create the judge record and the `HF_TOKEN` record reported by `required_secret_names`:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set HLE_JUDGE_API_KEY
osmosis secret set HF_TOKEN
```

Then add the judge record name to the HLE config's existing `[execution]` table:

```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[execution]
judge_api_key_secret = "HLE_JUDGE_API_KEY"
# judge_model = "openai/gpt-5.2"              # Optional override
```

`HF_TOKEN` is reserved by the benchmark runner for every benchmark, not only HLE. Never put it in top-level `[env]` or any `[agents.env]`, and never use it as a model's `api_key_secret`; for HLE, store the dataset credential only in the `HF_TOKEN` Platform secret record. A submit with a missing secret fails with the exact record names to create.

<Tip>
  Before submitting HLE, add `task_set = "parity"` under `[tasks]`. We recommend the published parity set for HLE runs; omitting `[tasks]` selects the full HLE benchmark.
</Tip>

## Submit the run

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark submit configs/benchmark/terminal-bench-smoke.toml
```

The confirmation preview shows the benchmark, task selection, agent models, attempts, concurrency, and secret scopes. To confirm non-interactively after reviewing the config:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json benchmark submit configs/benchmark/terminal-bench-smoke.toml --yes
```

The response includes:

* Generated run `id` and `name`
* Initial `status`
* Resolved `task_count`
* Submission timestamp in `created_at`
* `platform_url` for the benchmark run in the Platform

## Manage the run

Run lifecycle commands live under `benchmark runs`. List benchmark runs in the current workspace, which shows the same columns as the Platform's runs table, including each run's agent count and best pass\@1, then inspect the submitted run by name or ID:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs list
osmosis benchmark runs info <run-name>
```

`runs info` reports the run's status, progress, duration, and result totals, and scores every agent the way the run page does: rank, pass\@1 with its confidence interval, the deepest pass\@k, and per-task cost, time, and tokens.

Use `runs logs` to inspect lifecycle events and diagnose failures. Logs are returned oldest first within each page; pass the JSON response's `next_cursor` to page further back:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs logs <run-name>
osmosis --json benchmark runs logs <run-name> --cursor <next-cursor>
```

Stop a pending, queued, or running benchmark after reviewing the confirmation:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs stop <run-name>
```

Download the run summary and task-level results, or select additional output types:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Default: summary.csv and results.csv
osmosis benchmark runs download <run-name>

# Include every available export and result artifact
osmosis benchmark runs download <run-name> --type all

# Download only selected output types
osmosis benchmark runs download <run-name> --type summary,logs
```

Downloads use a run-scoped directory under `.osmosis/benchmarks/` by default and resume by skipping complete files. Available types are `summary`, `results`, `artifacts`, `logs`, and `all`. The layout is fixed:

```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
.osmosis/benchmarks/<run-name>/
├── summary.csv
├── results.csv
├── logs.txt
└── artifacts/<result-id>/<path>
```

Outputs are unavailable while a run is `pending` or `queued`. A `running` run downloads a current snapshot; pass `--overwrite` when refreshing files that may have changed without changing size.

You can also open the `platform_url` returned by `submit` or `runs info` to follow progress, compare agents, and inspect task-level results in the Platform.

## Expand the run

After the smoke run starts and produces expected results:

1. Add explicit `task_names`, select `categories`, use a published `parity` task set, or omit `[tasks]` for the full benchmark.
2. Increase `max_concurrent_attempts` within your workspace limits.
3. Add another `[[agents]]` table to compare harness or model combinations.
4. Review the new task and attempt counts before confirming the paid run.

<Warning>
  Benchmark runs can incur model and sandbox charges. Prefer a one-task smoke run before submitting a full benchmark, and do not pass `--yes` until the run scope has been reviewed.
</Warning>