> ## Documentation Index
> Fetch the complete documentation index at: https://docs.osmosis.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Benchmarks
> Add benchmarks, submit benchmark runs, and compare agents on the Osmosis platform
A benchmark is a published set of tasks with its own environment and grading. A benchmark run scores one or more agents on those tasks, and the platform owns the task environment, execution, and result collection, so you don't provision anything. Results land on a leaderboard that compares every agent you have run on that benchmark.
## Concepts
### Benchmark and Benchmark Run
A **benchmark** lives in your workspace and holds the task list, the harness and judge requirements, and the pass threshold. A **benchmark run** is one execution against a task selection. Runs are independent: submit as many as you need to compare agents, task subsets, or attempt budgets.
### Agent
An **agent** is a harness plus a model, for example `codex` with `openai/gpt-5.2`. One run can carry several agents, which is how you compare scaffolds or models under identical conditions. Every agent in a run gets the same tasks, the same attempt budget, and the same grading.
### Attempts and pass\@k
`attempts_per_task` sets how many independent tries each agent gets per task. Pass\@1 is the first-try rate; pass\@k is the rate of solving a task within k tries. Both are reported with a 95% confidence interval, and an agent the test can't separate from the best one shares rank 1 rather than being ranked below it. Hover its rank to see the comparison.
### Task Selection
Run every task, or narrow the selection by named task set, category, or explicit task names. Some benchmarks publish a **parity** task set, the sample their reference scores were measured on. HLE is the case to know: prefer its parity set when you want a number comparable with published results.
## Adding a Benchmark
Open **Benchmarks** in the sidebar. Osmosis-managed benchmarks are already in your workspace; **Add Benchmark** adds any dataset from the Harbor registry by name.
The table lists what your workspace can run:
| Column | Meaning |
| ------------ | -------------------------------------------------------------------------------------------------------------------- |
| **Name** | The benchmark; the row opens its page. |
| **Last Run** | The newest run's state, its age, and its name. Before the task list finishes syncing, this reports the sync instead. |
| **Tasks** | How many tasks the benchmark holds. Empty while the task list syncs, `unavailable` if the sync failed. |
| **Added** | When the benchmark entered the workspace. |
| **Added By** | Who added it. |
A Harbor benchmark's task list pages in from the registry after you add it, and runs can't be submitted until it is ready. If the sync fails, the row reports the reason and the benchmark's page offers **Retry sync**. Benchmarks you added can be removed again from the page's actions menu; managed ones cannot.
## The Benchmark Page
Each benchmark opens on its **Leaderboard** and its **Benchmark Runs** table, with **New Run** as the entry point for a submission. The header carries the benchmark's source reference (click to copy) and, once the task list is ready, a task-count badge; the actions menu offers **View source** and, after a failed sync, **Retry sync**.
### Leaderboard
One row per entrant, where an entrant is a harness-and-model pair. Each entrant is scored by its **latest** eligible run, so re-running an agent updates its standing instead of adding a row. The table ranks by Pass\@1, Pass\@k, Cost / task, Time / task, or Tokens / task. Click a metric's column header to re-rank; the ranked metric is carried in the URL. Ranks are competition ranks: tied entrants share a rank, so ranks can skip (1, 1, 3). Clicking a row opens the run behind that score.
An agent reaches the leaderboard when:
* its run finished, and the agent itself finished;
* the run covered the full task list, or a parity set the benchmark publishes for comparison;
* every task-and-attempt slot produced a result;
* the agent has a pass\@1 score.
Every eligible entrant ranks in the same list, sorted by the selected metric. Task set and benchmark version enter only the tie test: significance is checked between entrants that ran the same task set on the same resolved version, so a parity sample and a full run, or runs on different manifest versions, are never marked as tied, since they are not the same measurement.
A filtered run (a category or a handful of task names) is deliberately not ranked. It still gets a full run page, scores, and downloads.
## Submitting a Run
**New Run** opens a form with three tabs and a running summary of what you're about to submit:
* **Tasks**: every task, a named task set, categories, or explicit task names.
* **Agents**: one entry per agent, with its harness, its model, and the workspace or personal secret record holding that provider's API key. Add more entries to compare agents in one run.
* **Run settings**: attempts per task, concurrent attempts, timeout multiplier, retries, pass threshold, and the LLM judge when the benchmark scores with one.
API keys are always referenced by secret record name, never pasted into the form. Create the records first under **Secrets**; the form flags any the benchmark requires and your workspace is missing. Submission is where billing is checked, so a workspace without valid billing is told at submit rather than being locked out of the form.
Benchmark runs incur model and sandbox charges, and model spend lands on your own provider keys. Submit a one-task run first to confirm the agent and secrets work before committing to a full benchmark.
## Status Lifecycle
| Status | Description |
| ------------ | ---------------------------------------------------------- |
| **pending** | Submitted; the platform is preparing the run. |
| **queued** | Waiting on capacity to start. |
| **running** | Agents are working through their tasks; results stream in. |
| **finished** | Every expected result landed. Scores are final. |
| **failed** | The run stopped on an error. The Logs tab has the reason. |
| **stopped** | Someone stopped the run before it completed. |
## The Run Page
A run lives at `/benchmarks/runs/` and keeps a sidebar of status, progress, duration, tokens used, LLM cost, submission details, the pinned benchmark version, and its agents. Four tabs cover the run:
* **Overview**: **Agent Results** is a sortable table of the run's agents with the same metrics as the leaderboard, plus a progress column that keeps a live duration while the run is in progress. A pass\@k curve appears once agents have enough attempts, and a per-category breakdown once scored categories exist.
* **Task Results**: a searchable, filterable table of every task-and-attempt row, with the graded output, conversation, and artifacts for any result. The toolbar's Agent filter slices to one or more agents.
* **Configuration**: the resolved run configuration as TOML, including the benchmark version it was pinned to.
* **Logs**: lifecycle events from submission through cleanup.
**LLM Cost** is model spend on your own provider keys, reported by the harness. It is not billed by Osmosis.
Metrics, task-level results, and per-result artifacts all download from the run page. A pending or queued run has nothing to download yet; a running one downloads a snapshot.
## Stopping a Run
Pending, queued, and running runs can be stopped from the run page or its row in the runs table. The run moves to `stopped` once the platform finishes cleaning up its sandboxes. Results already ingested stay on the run.
## Next Steps
Submit and manage the same runs from a TOML config.
Reference for the benchmark TOML config.
Score your own rollout against a platform dataset.
Manage the secret records your agents reference.