Benchmarks

A benchmark is a published set of tasks with its own environment and grading. A benchmark run scores one or more agents on those tasks, and the platform owns the task environment, execution, and result collection, so you don’t provision anything. Results land on a leaderboard that compares every agent you have run on that benchmark.

​

Concepts

​

Benchmark and Benchmark Run

A benchmark lives in your workspace and holds the task list, the harness and judge requirements, and the pass threshold. A benchmark run is one execution against a task selection. Runs are independent: submit as many as you need to compare agents, task subsets, or attempt budgets.

​

Agent

An agent is a harness plus a model, for example codex with openai/gpt-5.2. One run can carry several agents, which is how you compare scaffolds or models under identical conditions. Every agent in a run gets the same tasks, the same attempt budget, and the same grading.

​

Attempts and pass@k

attempts_per_task sets how many independent tries each agent gets per task. Pass@1 is the first-try rate; pass@k is the rate of solving a task within k tries. Both are reported with a 95% confidence interval, and an agent the test can’t separate from the best one shares rank 1 rather than being ranked below it. Hover its rank to see the comparison.

​

Task Selection

Run every task, or narrow the selection by named task set, category, or explicit task names. Some benchmarks publish a parity task set, the sample their reference scores were measured on. HLE is the case to know: prefer its parity set when you want a number comparable with published results.

​

Adding a Benchmark

Open Benchmarks in the sidebar. Osmosis-managed benchmarks are already in your workspace; Add Benchmark adds any dataset from the Harbor registry by name. The table lists what your workspace can run: A Harbor benchmark’s task list pages in from the registry after you add it, and runs can’t be submitted until it is ready. If the sync fails, the row reports the reason and the benchmark’s page offers Retry sync. Benchmarks you added can be removed again from the page’s actions menu; managed ones cannot.

​

The Benchmark Page

Each benchmark opens on its Leaderboard and its Benchmark Runs table, with New Run as the entry point for a submission. The header carries the benchmark’s source reference (click to copy) and, once the task list is ready, a task-count badge; the actions menu offers View source and, after a failed sync, Retry sync.

​

Leaderboard

One row per entrant, where an entrant is a harness-and-model pair. Each entrant is scored by its latest eligible run, so re-running an agent updates its standing instead of adding a row. The table ranks by Pass@1, Pass@k, Cost / task, Time / task, or Tokens / task. Click a metric’s column header to re-rank; the ranked metric is carried in the URL. Ranks are competition ranks: tied entrants share a rank, so ranks can skip (1, 1, 3). Clicking a row opens the run behind that score. An agent reaches the leaderboard when:

• its run finished, and the agent itself finished;
• the run covered the full task list, or a parity set the benchmark publishes for comparison;
• every task-and-attempt slot produced a result;
• the agent has a pass@1 score.

Every eligible entrant ranks in the same list, sorted by the selected metric. Task set and benchmark version enter only the tie test: significance is checked between entrants that ran the same task set on the same resolved version, so a parity sample and a full run, or runs on different manifest versions, are never marked as tied, since they are not the same measurement.

​

Submitting a Run

New Run opens a form with three tabs and a running summary of what you’re about to submit:

• Tasks: every task, a named task set, categories, or explicit task names.
• Agents: one entry per agent, with its harness, its model, and the workspace or personal secret record holding that provider’s API key. Add more entries to compare agents in one run.
• Run settings: attempts per task, concurrent attempts, timeout multiplier, retries, pass threshold, and the LLM judge when the benchmark scores with one.

API keys are always referenced by secret record name, never pasted into the form. Create the records first under Secrets; the form flags any the benchmark requires and your workspace is missing. Submission is where billing is checked, so a workspace without valid billing is told at submit rather than being locked out of the form.

​

Status Lifecycle

​

The Run Page

A run lives at /benchmarks/runs/<run-id> and keeps a sidebar of status, progress, duration, tokens used, LLM cost, submission details, the pinned benchmark version, and its agents. Four tabs cover the run:

• Overview: Agent Results is a sortable table of the run’s agents with the same metrics as the leaderboard, plus a progress column that keeps a live duration while the run is in progress. A pass@k curve appears once agents have enough attempts, and a per-category breakdown once scored categories exist.
• Task Results: a searchable, filterable table of every task-and-attempt row, with the graded output, conversation, and artifacts for any result. The toolbar’s Agent filter slices to one or more agents.
• Configuration: the resolved run configuration as TOML, including the benchmark version it was pinned to.
• Logs: lifecycle events from submission through cleanup.

LLM Cost is model spend on your own provider keys, reported by the harness. It is not billed by Osmosis. Metrics, task-level results, and per-result artifacts all download from the run page. A pending or queued run has nothing to download yet; a running one downloads a snapshot.

​

Stopping a Run

Pending, queued, and running runs can be stopped from the run page or its row in the runs table. The run moves to stopped once the platform finishes cleaning up its sandboxes. Results already ingested stay on the run.

​

Next Steps