Benchmarks

A benchmark is a published task set with its own environment and grading. A benchmark run evaluates one or more agents while Osmosis manages execution and collects the results. Submit from the Platform’s New Run form or the CLI; eligible results appear on the benchmark leaderboard.

Benchmarks are in beta and enabled per account. If Benchmarks is unavailable in your workspace, contact us for access.

Concepts

Benchmark and Benchmark Run

A benchmark lives in your workspace and holds the task list, the harness and judge requirements, and the pass threshold. A benchmark run is one execution against a task selection. Runs are independent: submit as many as you need to compare agents, task subsets, or attempt budgets.

Agent

An agent is a harness plus a model, for example codex with openai/gpt-5.2. One run can carry several agents, which is how you compare scaffolds or models under identical conditions. Every agent in a run gets the same tasks, the same attempt budget, and the same grading.

Attempts and pass@k

attempts_per_task sets how many independent tries each agent gets per task. Pass@1 is the first-try rate; pass@k is the rate of solving a task within k tries. Both are reported with a 95% confidence interval, and an agent the test can’t separate from the best one shares rank 1 rather than being ranked below it. Hover its rank to see the comparison.

Task Selection

Run every task, or narrow the selection by named task set, category, or explicit task names. Some benchmarks publish a parity task set, the sample their reference scores were measured on. HLE is the case to know: prefer its parity set when you want a number comparable with published results.

Adding a Benchmark

Open Benchmarks in the sidebar. Osmosis-managed benchmarks are already in your workspace; Add Benchmark adds any dataset from the Harbor registry by name. Each card is a benchmark the workspace can run. Click a card to open its page. The card shows the name, a task-count badge once the task list is ready, and the description. Search filters the grid by name. A Harbor benchmark’s task list pages in from the registry after you add it, and runs can’t be submitted until it is ready. While it syncs, the card replaces the description with a progress bar. If the sync fails, the card shows the reason and the benchmark’s page offers Retry sync. Benchmarks you added can be removed again from the card’s actions menu; managed ones cannot. Any workspace member can add a benchmark, remove an added benchmark, and submit, stop, rename, or delete runs.

Managed Benchmarks

These are in every workspace and cannot be removed.

Benchmark	What it measures
Terminal-Bench 2.1	89 real terminal workflows: builds, debugging, data processing, security, and systems administration. Binary pass or fail per task.
SWE-Bench Pro	Long-horizon software engineering tasks from real repositories, verified by each task’s test suite. Independent audits estimate that a sizable share of tasks have defective reference patches, so treat absolute scores as directional and compare agents within a run.
DeepSWE	Repository-scale software engineering tasks scored on test transitions: `f2p` counts tests the agent fixed, `p2p` counts tests it kept passing, and `partial` grants credit for incomplete progress. The reward combines all three.
GDPVal	Real-world knowledge-work deliverables across 44 occupations, scored by an LLM judge against expert rubrics. Tasks without published expert deliverables are excluded from the run.
HLE	Humanity’s Last Exam: frontier academic reasoning questions graded by an LLM judge. The web leaderboard includes published results on the full 2,500-task set and platform runs on the 249-question parity set, but ranks each only against rows measured on the same task set.
BrowseComp	Hard-to-find web research questions. Runs default to the benchmark’s official one-shot scaffold, or use a harness for agentic browsing.
Toolathlon-Verified	Multi-service tool-use tasks that read and write real application state — email, documents, calendars, cloud storage — and verify the outcome. Runs use the official scaffold against Osmosis-managed accounts.

The Benchmark Page

Each benchmark opens on its Leaderboard and its Benchmark Runs table, with New Run as the entry point for a submission. The header carries the benchmark’s source reference (click to copy) and, once the task list is ready, a task-count badge; the actions menu offers View source and, after a failed sync, Retry sync.

Leaderboard

The web leaderboard combines platform runs with published reference scores. A platform entrant is a harness-and-model pair scored by its latest eligible run, so re-running an agent updates its standing instead of adding a row. Published rows carry origin and task-set badges plus a link to their source; because they do not belong to an Osmosis run, clicking them does not open a run page. Platform rows still open the run behind their score. The table ranks by Pass@1, Pass@k, Cost / task, Time / task, or Tokens / task. Click a metric’s column header to re-rank; the ranked metric is carried in the URL. Each row is ranked only against rows measured on the same full or parity task set. Ranks are competition ranks within that comparison group: tied entrants share a rank, so ranks can skip (1, 1, 3). An agent reaches the leaderboard when:

its run finished, and the agent itself finished;
the run covered the full task list, or a parity set the benchmark publishes for comparison;
every task-and-attempt slot produced a result;
the agent has a pass@1 score.

Platform rows and published rows appear in one table, but full-set and parity-set measurements remain separate ranking groups. For platform runs, significance is checked only between entrants that used the same task set on the same resolved benchmark version. HLE’s published full 2,500-task results therefore remain separate from platform runs on its 249-question parity set.

A filtered run (a category or a handful of task names) is deliberately not ranked. It still gets a full run page, scores, and downloads.

Inspecting a Benchmark from the CLI

List the benchmarks available in the current workspace, then inspect the one you plan to run:

osmosis benchmark list
osmosis benchmark info terminal-bench-2-1

benchmark info shows the key and exact name, a source_url linking to the upstream project (or the adapter repository when the benchmark is adapted), task and category counts, named task sets, harness and judge requirements, and pass threshold, followed by the platform leaderboard and the workspace’s runs on it. The CLI output remains platform-only; published reference rows are shown on the web leaderboard. When a benchmark’s published scores were measured on a specific harness, that harness is reported as the default. Terminal-Bench 2.1’s is terminus-2, and a run on another harness is not comparable with those scores. Use JSON output to inspect the complete task manifest before choosing task_names or categories:

osmosis --json benchmark info terminal-bench-2-1

Every task in the JSON manifest includes difficulty, whose value is easy, medium, hard, or null. A null value means the benchmark source did not provide a difficulty; do not infer one. For HLE, the command marks parity as the recommended named task set. Omit [tasks] to select the full benchmark. While a Harbor registry benchmark’s task list is still syncing, the Last Run column reports the sync and its task progress, and submitting against it fails until it completes. If the sync failed, benchmark list reports the reason there and unavailable under Tasks, with a sync_error in JSON output. The reported platform_url opens the benchmark’s page, with its leaderboard, runs, and Retry sync.

Submitting a Run from the Platform

New Run opens a form with three tabs and a running summary of what you’re about to submit:

Tasks: every task, a named task set, categories, or explicit task names.
Agents: one entry per agent, with its harness, its model, and the workspace or personal secret record holding that provider’s API key. Add more entries to compare agents in one run. Each agent also takes its own environment variables as name-and-value rows.
Run settings: attempts per task, concurrent attempts, timeout multiplier, retries, pass threshold, the LLM judge when the benchmark scores with one, and any secrets this run supplies itself.

The model field, and the judge model field when the benchmark uses one, search the models the platform can route and group them by provider. A slug that isn’t in the list can still be typed in, so a model released after the catalog was built is never blocked. Custom-endpoint agents keep a free-text model field, since that name belongs to the server you point at. Submission is where billing is checked, so a workspace without valid billing is told at submit rather than being locked out of the form.

How the Form Gets Credentials

API keys are referenced by secret record name, never pasted into a model or judge field. Create the records first under Secrets and pick them by name. Two sections in Run settings cover credentials the model and judge fields don’t name:

Verifier secrets, on benchmarks added from a Harbor registry, name the records a dataset’s own LLM verifier reads. Each is delivered to the verifier under its record name and never reaches the agent. Managed benchmarks model their verifier credentials themselves and don’t offer this section.
Run secrets are supplied with a single run as a name and a value. The value is not added to the Osmosis secret store or persistent run config: the run records only the name, under the Run scope, and a run secret shadows a stored record of the same name for that run. Re-running from a previous run brings the names back with empty values for you to re-enter.

Benchmark runs incur model and sandbox charges, and model spend lands on your own provider keys. Submit a one-task run first to confirm the agent and secrets work before committing to a full benchmark.

Submitting a Run from the CLI

The CLI submits an Osmosis TOML config from configs/benchmark/ in the synced workspace repository.

Add the benchmark to the workspace first (see Adding a Benchmark). [experiment].benchmark resolves a benchmark already in the workspace by key, name, or ID, not a local dataset path.

Create a Config

Copy the workspace template and edit it:

cp configs/benchmark/default.toml configs/benchmark/terminal-bench-smoke.toml

For the first run, select one task and one agent:

configs/benchmark/terminal-bench-smoke.toml

[experiment]
benchmark = "terminal-bench-2-1"

[tasks]
task_names = ["terminal-bench/git-multibranch"]

[[agents]]
harness = "codex"

[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"

[execution]
attempts_per_task = 1
max_concurrent_attempts = 1
timeout_multiplier = 1.0
max_retries = 0

See Configuration Files for provider, endpoint, hosted model, task-filter, and execution fields.

Register Credentials

api_key_secret, harness_api_key_secret, and judge_api_key_secret contain Platform secret record names, never credential values. Create every record referenced by the config before submitting:

osmosis secret set OPENAI_API_KEY

Personal scope is the default. Use --scope workspace for a credential shared with workspace members who can submit runs. Cursor CLI authenticates separately from the model. Set that agent’s harness_api_key_secret to CURSOR_API_KEY. Any other value is rejected at submit. Mini SWE-agent always rejects harness_api_key_secret. With provider and endpoint models, the platform reuses the model’s api_key_secret and injects it as MSWEA_API_KEY, so that name cannot also appear in the agent’s literal environment. Hosted models receive no injected model key and may set MSWEA_API_KEY explicitly.

osmosis secret set CURSOR_API_KEY

Harnesses other than cursor-cli reject harness_api_key_secret. See Configuration Files for a complete agent example. HLE and GDPVal use an LLM judge and require judge_api_key_secret. Set it to a Platform secret record name; judge_model is optional and uses the benchmark default when omitted. For HLE, create the judge record:

osmosis secret set HLE_JUDGE_API_KEY

Then add the judge record name to the HLE config’s existing [execution] table:

[execution]
judge_api_key_secret = "HLE_JUDGE_API_KEY"
# judge_model = "openai/gpt-5.2"              # Optional override

A submit with a missing secret fails with the exact record names to create.

Before submitting HLE, add task_set = "parity" under [tasks]. We recommend the published parity set for HLE runs; omitting [tasks] selects the full HLE benchmark.

Submit the Run

osmosis benchmark submit configs/benchmark/terminal-bench-smoke.toml

The confirmation preview shows the benchmark, task selection, agent models, attempts, concurrency, and secret scopes. To confirm non-interactively after reviewing the config:

osmosis --json benchmark submit configs/benchmark/terminal-bench-smoke.toml --yes

The response includes:

Generated run id and name
Initial status
Resolved task_count
Submission timestamp in created_at
platform_url for the benchmark run in the Platform

Status Lifecycle

Status	Description
pending	Submitted; the platform is preparing the run.
queued	Waiting on capacity to start.
running	Agents are working through their tasks; results stream in.
finished	Every expected result landed. Scores are final.
failed	The run stopped on an error. The Logs tab has the reason.
stopped	Someone stopped the run before it completed.

Monitoring

The Run Page

A run lives at /benchmarks/runs/<run-id> and keeps a sidebar of status, progress, duration, tokens used, LLM cost, submission details, the pinned benchmark version, and its agents. Four tabs cover the run:

Overview: Agent Results is a sortable table of the run’s agents with the same metrics as the leaderboard, plus a progress column that keeps a live duration while the run is in progress. A pass@k curve appears once agents have enough attempts, and a per-category breakdown once scored categories exist.
Task Results: a searchable, filterable table of every task-and-attempt row, with the graded output, conversation, and artifacts for any result. The toolbar’s Agent filter slices to one or more agents.
Configuration: the resolved run configuration as TOML, including the benchmark version it was pinned to.
Logs: lifecycle events from submission through cleanup.

LLM Cost is model spend on your own provider keys, reported by the harness. It is not billed by Osmosis. Open the platform_url returned by submit or runs info to follow progress, compare agents, and inspect task-level results here.

CLI Commands

Run lifecycle commands live under benchmark runs. List benchmark runs in the current workspace, which shows the same columns as the Platform’s runs table, including each run’s agent count and best pass@1, then inspect the submitted run by name or ID:

osmosis benchmark runs list
osmosis benchmark runs info <run-name>

runs info reports the run’s status, progress, duration, and result totals, and scores every agent the way the run page does: rank, pass@1 with its confidence interval, the deepest pass@k, and per-task cost, time, and tokens. Use runs logs to inspect lifecycle events and diagnose failures. Logs are returned oldest first within each page; pass the JSON response’s next_cursor to page further back:

osmosis benchmark runs logs <run-name>
osmosis --json benchmark runs logs <run-name> --cursor <next-cursor>

Downloading Outputs

Metrics, task-level results, and per-result artifacts all download from the run page. From the CLI, download the run summary and task-level results, or select additional output types:

# Default: summary.csv and results.csv
osmosis benchmark runs download <run-name>

# Include every available export and result artifact
osmosis benchmark runs download <run-name> --type all

# Download only selected output types
osmosis benchmark runs download <run-name> --type summary,logs

Downloads use a run-scoped directory under .osmosis/benchmarks/ by default and resume by skipping complete files. Available types are summary, results, artifacts, logs, and all. The layout is fixed:

.osmosis/benchmarks/<run-name>/
├── summary.csv
├── results.csv
├── logs.txt
└── artifacts/<result-id>/<path>

Outputs are unavailable while a run is pending or queued. A running run downloads a current snapshot; pass --overwrite when refreshing files that may have changed without changing size.

Stopping a Run

Pending, queued, and running runs can be stopped from the run page or its row in the runs table. The run moves to stopped once the platform finishes cleaning up its sandboxes. Results already ingested stay on the run. From the CLI, stop a run after reviewing the confirmation:

osmosis benchmark runs stop <run-name>

Expanding a Run

After the smoke run starts and produces expected results:

Add explicit task_names, select categories, use a published parity task set, or omit [tasks] for the full benchmark.
Increase max_concurrent_attempts within your workspace limits.
Add another [[agents]] table to compare harness or model combinations.
Review the new task and attempt counts before confirming the paid run.

Next Steps

CLI Benchmarks

Discover benchmarks and submit, inspect, or download runs from the CLI.

Config Files

Reference for the benchmark TOML config.

Evaluation Runs

Score your own rollout against a platform dataset.

Secrets

Manage the secret records your agents reference.

Get started

Runs

Resources

Workspace

Concepts

Benchmark and Benchmark Run

Agent

Attempts and pass@k

Task Selection

Adding a Benchmark

Managed Benchmarks

The Benchmark Page

Leaderboard

Inspecting a Benchmark from the CLI

Submitting a Run from the Platform

How the Form Gets Credentials

Submitting a Run from the CLI

Create a Config

Register Credentials

Submit the Run

Status Lifecycle

Monitoring

The Run Page

CLI Commands

Downloading Outputs

Stopping a Run

Expanding a Run

Next Steps

CLI Benchmarks

Config Files

Evaluation Runs

Secrets

​Concepts

​Benchmark and Benchmark Run

​Agent

​Attempts and pass@k

​Task Selection

​Adding a Benchmark

​Managed Benchmarks

​The Benchmark Page

​Leaderboard

​Inspecting a Benchmark from the CLI

​Submitting a Run from the Platform

​How the Form Gets Credentials

​Submitting a Run from the CLI

​Create a Config

​Register Credentials

​Submit the Run

​Status Lifecycle

​Monitoring

​The Run Page

​CLI Commands

​Downloading Outputs

​Stopping a Run

​Expanding a Run

​Next Steps

CLI Benchmarks

Config Files

Evaluation Runs

Secrets

Concepts

Benchmark and Benchmark Run

Agent

Attempts and pass@k

Task Selection

Adding a Benchmark

Managed Benchmarks

The Benchmark Page

Leaderboard

Inspecting a Benchmark from the CLI

Submitting a Run from the Platform

How the Form Gets Credentials

Submitting a Run from the CLI

Create a Config

Register Credentials

Submit the Run

Status Lifecycle

Monitoring

The Run Page

CLI Commands

Downloading Outputs

Stopping a Run

Expanding a Run

Next Steps