# Changelog
Source: https://docs.osmosis.ai/changelog
Follow user-visible changes to the Osmosis Python SDK and CLI
This changelog highlights the SDK and CLI changes that affect how you install,
build, evaluate, and train rollouts. It is more task-oriented than the canonical
[SDK changelog](https://github.com/Osmosis-AI/osmosis-sdk-python/blob/main/CHANGELOG.md),
which remains the complete repository record.
Release candidates remain in this timeline as incremental release records.
Each stable release provides a complete summary from the previous stable
version, so you do not need to read every release-candidate entry before
upgrading.
Version `0.3.0rc1` introduces the 0.3 rollout protocol. A workflow execution
now produces exactly one `RolloutSample`, and its grader assigns exactly one
reward.
### What changed
* `GraderContext.samples` is now `GraderContext.sample`, and
`set_sample_reward(sample_id, reward)` is now `set_reward(reward)`.
* Custom integrations register one source with `set_sample_source()` and read
it with `get_sample()`.
* `RolloutSample.id` and `MultiTurnMode` were removed. The rollout URL now
supplies execution identity.
* Model and callback requests use rollout-scoped URLs. Integrations no longer
attach per-call sample or rollout routing headers.
* Backends exchange `sample.json` and a single-value
`reward.json` (`{"reward": }`).
* Strands and OpenAI Agents integrations enforce one registered agent or
session per workflow execution.
### Who needs to act
Update custom graders, custom sample sources, backend adapters, or workflows
that create more than one registered Strands agent or OpenAI Agents session.
Evaluation and training can still request multiple independent executions for
the same prompt.
Follow [Migrate from v0.2 to v0.3](/migration-guides/v0-3) for before-and-after
code and verification steps.
[SDK changelog](https://github.com/Osmosis-AI/osmosis-sdk-python/blob/main/CHANGELOG.md#030rc1---2026-07-28)
· [GitHub release](https://github.com/Osmosis-AI/osmosis-sdk-python/releases/tag/v0.3.0rc1)
· [Full diff](https://github.com/Osmosis-AI/osmosis-sdk-python/compare/v0.2.30...v0.3.0rc1)
# Benchmark Runs
Source: https://docs.osmosis.ai/cli/benchmark-runs
Configure, submit, monitor, and download managed benchmark runs from the Osmosis CLI
Benchmark runs compare one or more agent harness and model combinations on a benchmark managed by the Platform. The CLI submits an Osmosis TOML config; the Platform owns the benchmark source, task environment, execution, and result collection.
Add the benchmark to the workspace in the Platform before submitting it from the CLI. `[experiment].benchmark` resolves a benchmark already in the workspace by key, name, or ID, not a local dataset path.
## Inspect the benchmark
List the benchmarks available in the current workspace, then inspect the one you plan to run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark list
osmosis benchmark info terminal-bench-2-1
```
`benchmark info` shows the key and exact name, a `source_url` linking to the upstream project (or the adapter repository when the benchmark is adapted), task and category counts, named task sets, harness and judge requirements, pass threshold, and any implicit required secret record names, followed by the benchmark's leaderboard and the workspace's runs on it. When a benchmark's published scores were measured on a specific harness, that harness is reported as the default. Terminal-Bench 2.1's is `terminus-2`, and a run on another harness is not comparable with those scores. Use JSON output to inspect the complete task manifest and `required_secret_names` before choosing `task_names` or `categories`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json benchmark info terminal-bench-2-1
```
Every task in the JSON manifest includes `difficulty`, whose value is `easy`, `medium`, `hard`, or `null`. A `null` value means the benchmark source did not provide a difficulty; do not infer one.
For HLE, the command marks `parity` as the recommended named task set. Omit `[tasks]` to select the full benchmark.
A Harbor registry benchmark's task list pages in from the registry after the benchmark is added. While that is in progress the Last Run column reports the sync and its task progress, and submitting against it fails until it completes. If the sync failed, `benchmark list` reports the reason there and `unavailable` under Tasks, with a `sync_error` in JSON output; retry it from the benchmark's page in the Platform. The reported `platform_url` opens that page, with its leaderboard and runs.
## Create a config
Copy the workspace template and edit it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
cp configs/benchmark/default.toml configs/benchmark/terminal-bench-smoke.toml
```
For the first run, select one task and one agent:
```toml configs/benchmark/terminal-bench-smoke.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
benchmark = "terminal-bench-2-1"
[tasks]
task_names = ["terminal-bench/git-multibranch"]
[[agents]]
harness = "codex"
[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"
[execution]
attempts_per_task = 1
max_concurrent_attempts = 1
timeout_multiplier = 1.0
max_retries = 0
```
See [Configuration Files](/cli/config-files#benchmark-config) for provider, endpoint, hosted model, task-filter, and execution fields.
## Register credentials
`api_key_secret`, `harness_api_key_secret`, and `judge_api_key_secret` contain Platform secret record names, never credential values. Create every record referenced by the config before submitting:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set OPENAI_API_KEY
```
Also create every record named in the `benchmark info` response's `required_secret_names`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set NAME
```
These are implicit benchmark requirements and are not repeated in the TOML. The response returns record names only. For example, HLE includes `HF_TOKEN`.
Personal scope is the default. Use `--scope workspace` for a credential shared with workspace members who can submit runs.
Cursor CLI and Mini SWE-agent authenticate separately from the model. Set that agent's `harness_api_key_secret` to `CURSOR_API_KEY` for Cursor CLI, or to `MSWEA_API_KEY` for Mini SWE-agent. Those are the variables the harnesses read, and any other value is rejected at submit.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set CURSOR_API_KEY
osmosis secret set MSWEA_API_KEY
```
Configure only the record needed by the selected harness. Harnesses other than `cursor-cli` and `mini-swe-agent` reject `harness_api_key_secret`. See [Configuration Files](/cli/config-files#agents-and-agentsmodel) for a complete agent example.
HLE and GDPVal use an LLM judge and require `judge_api_key_secret`. Set it to a Platform secret record name; `judge_model` is optional and uses the benchmark default when omitted. For HLE, create the judge record and the `HF_TOKEN` record reported by `required_secret_names`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set HLE_JUDGE_API_KEY
osmosis secret set HF_TOKEN
```
Then add the judge record name to the HLE config's existing `[execution]` table:
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[execution]
judge_api_key_secret = "HLE_JUDGE_API_KEY"
# judge_model = "openai/gpt-5.2" # Optional override
```
`HF_TOKEN` is reserved by the benchmark runner for every benchmark, not only HLE. Never put it in top-level `[env]` or any `[agents.env]`, and never use it as a model's `api_key_secret`; for HLE, store the dataset credential only in the `HF_TOKEN` Platform secret record. A submit with a missing secret fails with the exact record names to create.
Before submitting HLE, add `task_set = "parity"` under `[tasks]`. We recommend the published parity set for HLE runs; omitting `[tasks]` selects the full HLE benchmark.
## Submit the run
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark submit configs/benchmark/terminal-bench-smoke.toml
```
The confirmation preview shows the benchmark, task selection, agent models, attempts, concurrency, and secret scopes. To confirm non-interactively after reviewing the config:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json benchmark submit configs/benchmark/terminal-bench-smoke.toml --yes
```
The response includes:
* Generated run `id` and `name`
* Initial `status`
* Resolved `task_count`
* Submission timestamp in `created_at`
* `platform_url` for the benchmark run in the Platform
## Manage the run
Run lifecycle commands live under `benchmark runs`. List benchmark runs in the current workspace, which shows the same columns as the Platform's runs table, including each run's agent count and best pass\@1, then inspect the submitted run by name or ID:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs list
osmosis benchmark runs info
```
`runs info` reports the run's status, progress, duration, and result totals, and scores every agent the way the run page does: rank, pass\@1 with its confidence interval, the deepest pass\@k, and per-task cost, time, and tokens.
Use `runs logs` to inspect lifecycle events and diagnose failures. Logs are returned oldest first within each page; pass the JSON response's `next_cursor` to page further back:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs logs
osmosis --json benchmark runs logs --cursor
```
Stop a pending, queued, or running benchmark after reviewing the confirmation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs stop
```
Download the run summary and task-level results, or select additional output types:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Default: summary.csv and results.csv
osmosis benchmark runs download
# Include every available export and result artifact
osmosis benchmark runs download --type all
# Download only selected output types
osmosis benchmark runs download --type summary,logs
```
Downloads use a run-scoped directory under `.osmosis/benchmarks/` by default and resume by skipping complete files. Available types are `summary`, `results`, `artifacts`, `logs`, and `all`. The layout is fixed:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
.osmosis/benchmarks//
├── summary.csv
├── results.csv
├── logs.txt
└── artifacts//
```
Outputs are unavailable while a run is `pending` or `queued`. A `running` run downloads a current snapshot; pass `--overwrite` when refreshing files that may have changed without changing size.
You can also open the `platform_url` returned by `submit` or `runs info` to follow progress, compare agents, and inspect task-level results in the Platform.
## Expand the run
After the smoke run starts and produces expected results:
1. Add explicit `task_names`, select `categories`, use a published `parity` task set, or omit `[tasks]` for the full benchmark.
2. Increase `max_concurrent_attempts` within your workspace limits.
3. Add another `[[agents]]` table to compare harness or model combinations.
4. Review the new task and attempt counts before confirming the paid run.
Benchmark runs can incur model and sandbox charges. Prefer a one-task smoke run before submitting a full benchmark, and do not pass `--yes` until the run scope has been reviewed.
# Command Reference
Source: https://docs.osmosis.ai/cli/command-reference
Review all Osmosis CLI commands
Run `osmosis -h` to see all available commands. Every sub-command supports `-h` / `--help`.
The global output flags `--json` and `--plain` are position-independent — they work before or after the command:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json
osmosis --json
```
Use `--json` for automation and AI agents. Use `--plain` for low-noise shell output.
***
## auth
Manage CLI authentication.
| Command | Description |
| --------------------- | ------------------------------------------------------------------- |
| `osmosis auth login` | Authenticate via browser OAuth or personal access token |
| `osmosis auth logout` | Revoke the current session and clear local credentials |
| `osmosis auth whoami` | Show the authenticated user, token expiration, and linked workspace |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis auth login
osmosis auth whoami
```
When you run `osmosis auth whoami` from inside a workspace repository, the CLI also resolves the linked Platform workspace. `--json` reports it as a `workspace` object (`id`, `name`, `role`); other modes show `Workspace` and `Role` rows. Outside a workspace repository, or when offline or logged out, `workspace` is `null` and the command still works.
***
## doctor
Inspect and optionally repair the workspace directory scaffold.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor [path] [--fix]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ------ | ------- | ---------------------------------------------------------------------- |
| `path` | `path` | `.` | Path inside the workspace directory |
| `--fix` | flag | — | Create missing scaffold directories without overwriting existing files |
When you are logged in and the workspace directory has a Platform-connected `origin` remote, `doctor` also reports the linked workspace (`Linked workspace: `, or the `workspace` resource field in `--json`). The lookup is best-effort — offline or logged out, the field is `null` and `doctor` still works.
***
## template
Add starter rollouts from the Osmosis workspace template catalog.
### template list
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template list
```
Lists available starter templates.
### template apply
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template apply [--force]
```
| Argument / Option | Type | Description |
| ----------------- | ---------------- | ------------------------------------------------------------- |
| `name` | `str` (required) | Template name from `osmosis template list` |
| `-f`, `--force` | flag | Replace the template-owned rollout directory and config files |
***
## rollout
Create local rollout scaffolds and list synced rollouts.
### rollout init
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis rollout init [--force]
```
Creates:
* `rollouts//main.py`
* `rollouts//pyproject.toml`
* `rollouts//README.md`
* `configs/eval/.toml`
* `configs/training/.toml`
| Argument / Option | Type | Description |
| ----------------- | ---------------- | --------------------------------------------------------- |
| `name` | `str` (required) | Lowercase rollout name using letters, digits, and hyphens |
| `-f`, `--force` | flag | Replace existing scaffold paths for this rollout |
### rollout list
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis rollout list [--limit N] [--all] [--branch NAME]
```
| Option | Type | Default | Description |
| ---------- | ----- | ------------------ | ------------------------------------- |
| `--limit` | `int` | `50` | Maximum number of rollouts to show |
| `--all` | flag | — | Show all rollouts |
| `--branch` | `str` | Repository default | Show rollouts synced from this branch |
***
## eval
Submit and manage evaluation runs, and run LLM-as-judge rubric scoring locally.
### eval submit
Submit an evaluation run from a TOML config under `configs/eval/`. The platform clones the workspace repository identified by the `origin` remote and runs the rollout server-side, so push your commits before submitting.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit [--yes]
```
| Argument / Option | Type | Description |
| ----------------- | ----------------- | ------------------------------- |
| `config_path` | `path` (required) | Eval TOML under `configs/eval/` |
| `-y`, `--yes` | flag | Skip the confirmation prompt |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit configs/eval/my-rollout.toml
```
Config values come from the local TOML file. Rollout code comes from the synced workspace repository. Use `[experiment].branch` or `[experiment].commit_sha` to select the source; they are mutually exclusive.
### eval list
List evaluation runs for the current workspace repository, including each run's status, average reward, and progress.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval list [--limit N] [--all]
```
| Option | Type | Default | Description |
| --------- | ----- | ------- | ----------------------------------------- |
| `--limit` | `int` | `50` | Maximum number of evaluation runs to show |
| `--all` | flag | — | Show all evaluation runs |
### eval info
Show details, results, and metrics for an evaluation run.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval info [--output PATH]
```
| Argument / Option | Type | Description |
| ----------------- | ---------------- | ------------------------------------------------------------------- |
| `name_or_id` | `str` (required) | Evaluation run name or ID |
| `-o`, `--output` | `path` | Run output root. Defaults to `.osmosis/evals//` in rich mode. |
The sidebar shows status, progress (rows completed and percent), and duration. Below the sidebar, the output includes:
* **Summary** — pass rate, tokens used, reward stats (including mean), and per-status row counts.
* **Configuration** — entrypoint, commit SHA, dataset stats, pass thresholds, pass\@k, token limits, timeouts, `[env]` keys, and resolved secret scopes.
* **Results** — per-row reward, pass/fail, and the most recent platform logs.
For completed runs, `eval info` also fetches metrics (duration, pass rate, tokens used, reward statistics, and pass\@k) from the platform and exports them as JSON:
* In rich mode, the CLI saves metrics to `.osmosis/evals//metrics.json` by default when metrics are available.
* In JSON or plain mode, pass `-o` / `--output` to set the run output root; the CLI writes `metrics.json` inside that directory.
* The CLI creates parent directories automatically.
* If metrics are not yet available (for example, the run is still pending), the CLI prints a notice instead of writing a file.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval info my-eval-run
osmosis --json eval info my-eval-run -o ./eval-outputs/
```
`-o` / `--output` now points at the run output root, not a metrics filename. Legacy files under `.osmosis/metrics/` are left in place — delete them manually once you have re-run `eval info` under the new layout.
### eval download
Download evaluation run outputs — metrics, trajectories, artifacts, and logs — from the platform to your local disk.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval download [--type TYPES] [--rows ROWS] [--output PATH] [--overwrite] [--yes]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `name_or_id` | `str` (required) | — | Evaluation run name or ID |
| `--type` | `str` | `metrics,trajectories` | Comma-separated selector: `metrics`, `trajectories`, `artifacts`, `logs`, or `all`. Replaces the default selection. |
| `--rows` | `str` | — | Rows to include for trajectories and artifacts, for example `3,7,10-20`. Each selected row includes all of its runs. |
| `-o`, `--output` | `path` | `.osmosis/evals//` | Run output root. Filenames and subdirectories below the root are fixed. |
| `--overwrite` | flag | — | Re-download files that already exist locally with a matching size. |
| `-y`, `--yes` | flag | — | Skip the confirmation prompt for downloads larger than 100 MiB. |
Downloads land in a fixed run-scoped layout so re-running the command resumes cleanly:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
.osmosis/evals//
├── metrics.json
├── summary.jsonl
├── trajectories/row__run_.json
├── artifacts/row__run_/...
└── logs.txt
```
Behavior:
* The CLI skips files whose local size matches the platform manifest unless you pass `--overwrite`, so re-running the command only fetches missing or partial files.
* Downloads over 100 MiB in total require confirmation; pass `--yes` to skip the prompt in scripts.
* The CLI requests presigned URLs in bounded batches and downloads up to eight files concurrently. Each file writes to a `*.partial` sibling, then moves into place atomically once the CLI verifies its size.
* The CLI retries failed files with backoff. If some files still fail, it reports each failed path and exits partial — re-run the command to retry only what is missing.
* Pending runs fail early with a clear message; the platform performs authoritative path validation.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Default: metrics + trajectories for the whole run
osmosis eval download my-eval-run
# Only artifacts for rows 3, 7, and 10–20
osmosis eval download my-eval-run --type artifacts --rows 3,7,10-20
# Everything, into a custom root, non-interactive
osmosis eval download my-eval-run --type all -o ./eval-outputs/ --yes
```
### eval stop
Stop a pending or running evaluation run.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval stop [--yes]
```
| Argument / Option | Type | Description |
| ----------------- | ---------------- | ---------------------------- |
| `name_or_id` | `str` (required) | Evaluation run name or ID |
| `-y`, `--yes` | flag | Skip the confirmation prompt |
### eval logs
Show the most recent lifecycle logs for an evaluation run, oldest first. Use this to diagnose failed runs.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval logs [--limit N] [--cursor CURSOR]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ------- | ------------------------------------------- |
| `name_or_id` | `str` (required) | — | Evaluation run name or ID |
| `--limit` | `int` | `50` | Entries per page (1–200) |
| `--cursor` | `str` | — | Cursor from a previous page's `next_cursor` |
When older entries exist, `--json` output includes a non-null `next_cursor`. Pass it back with `--cursor` to page further back in time.
### eval rubric
Run LLM-as-judge evaluation locally over a JSONL conversation file. This sub-command does not require a workspace directory or platform authentication and does not run a rollout.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval rubric -d -r --model
```
| Option | Type | Default | Description |
| ---------------- | ---------------- | ------- | ------------------------------------ |
| `-d`, `--data` | `str` (required) | — | Path to JSONL conversations |
| `-r`, `--rubric` | `str` (required) | — | Rubric text or `@file.txt` |
| `--model` | `str` (required) | — | Judge model in LiteLLM format |
| `-n`, `--number` | `int` | `1` | Number of evaluation runs per record |
| `-o`, `--output` | `str` | — | Path for JSON output |
| `--api-key` | `str` | — | API key for the judge model |
| `--timeout` | `float` | — | Request timeout in seconds |
| `--score-min` | `float` | `0.0` | Minimum score |
| `--score-max` | `float` | `1.0` | Maximum score |
***
## benchmark
Discover managed benchmarks and submit, inspect, and manage benchmark runs for the current workspace. `benchmark list` and `benchmark info` act on benchmarks themselves: the workspace list and one benchmark's page. Commands for an individual run live under `benchmark runs`.
### benchmark list
List the benchmarks added to the current workspace.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark list [--limit N] [--all]
```
| Option | Type | Default | Description |
| --------- | ----- | ------- | ------------------------------------ |
| `--limit` | `int` | `50` | Maximum number of benchmarks to show |
| `--all` | flag | off | Fetch every benchmark |
The table mirrors the Platform's benchmarks page: Name, the shell-safe Key, Last Run, Tasks, and Added By. Last Run reads as the newest run's state, its age, and its name (`Finished · 2d ago · brave-otter`). The key, the exact case-sensitive name, and the ID all work in `[experiment].benchmark`. JSON list items include `run_count`, `running_count`, `last_run_at`, `last_run_status`, `last_run_name`, and `creator_name`.
A Harbor registry benchmark's task list pages in from the registry after the benchmark is added. Until it finishes, Last Run reports the sync instead of a run (`Syncing · 412 / 500 tasks`), Tasks stays empty, and a run cannot be submitted against it. A benchmark whose sync failed reports the reason in Last Run, shows `unavailable` under Tasks, and returns a `sync_error`; retry its sync from the benchmark's page in the Platform. The reported `platform_url` opens that page, with its leaderboard and runs.
### benchmark info
Show a benchmark's summary, leaderboard, and runs.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark info [--limit N] [--all]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ------- | ------------------------------------------------ |
| `key_name_or_id` | `str` (required) | — | Benchmark key, exact name, or ID |
| `--limit` | `int` | `50` | Maximum number of runs to show in the runs table |
| `--all` | flag | off | Show every run in the runs table |
The rich output opens with the benchmark summary (the source's upstream page, runner, task and category counts, named task sets, harness and judge requirements, pass threshold, and implicit required secret record names), followed by the benchmark's leaderboard and a table of the workspace's runs on it. `--limit` and `--all` apply to the runs table. When a benchmark's published scores were measured on a specific harness, that harness is reported as the default. JSON output also includes the complete task manifest, `required_secret_names`, and a `leaderboard` array:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --json benchmark info terminal-bench-2-1
```
Each `leaderboard` entry carries the rank, a `tied` flag (set when the paired test can't separate that entrant from the top-ranked one, so it shares rank 1), the task set (`full` or `parity`), the harness and model, `pass_at_1` with its confidence interval, `pass_at_k`, the per-task `cost_per_task`, `mean_duration_seconds`, and `tokens_per_task`, and the scoring run's `id`, `name`, and `platform_url`. Those are the same five metrics the Platform's leaderboard ranks by.
Each task in the JSON manifest has a `difficulty` field with the value `easy`, `medium`, `hard`, or `null`. `null` means the source did not provide a difficulty, and clients must not infer one.
`required_secret_names` contains record names only. Before submission, ensure every listed record exists with `osmosis secret set NAME`; these implicit requirements are not repeated in the benchmark TOML. For example, HLE lists `HF_TOKEN`.
For HLE, `benchmark info` also marks `parity` as the recommended named task set. Omit `[tasks]` from the config to select the full benchmark; use `task_names`, `categories`, or a listed `task_set` for a subset.
### benchmark submit
Submit a benchmark run from an Osmosis TOML config under `configs/benchmark/`.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark submit [--yes]
```
| Argument / Option | Type | Description |
| ----------------- | ----------------- | ----------------------------------------- |
| `config_path` | `path` (required) | Benchmark TOML under `configs/benchmark/` |
| `-y`, `--yes` | flag | Skip the confirmation prompt |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark submit configs/benchmark/terminal-bench-smoke.toml
```
The benchmark named in `[experiment].benchmark` must already be added to the current workspace. The command previews the selected tasks, agents, attempts, concurrency, and resolved secret scopes before submission. The result includes the generated run name, task count, status, and `platform_url`.
See [Benchmark Runs](/cli/benchmark-runs) for the workflow and [Configuration Files](/cli/config-files#benchmark-config) for the full TOML schema.
### benchmark runs list
List benchmark runs for the current workspace. The table matches the Platform's runs table: Name, Status, Progress, Benchmark, Agents, Best Pass\@1, Submitted, and Submitted By.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs list [--limit N] [--all]
```
| Option | Type | Default | Description |
| --------- | ----- | ------- | ---------------------------------------- |
| `--limit` | `int` | `50` | Maximum number of benchmark runs to show |
| `--all` | flag | — | Show all benchmark runs |
### benchmark runs info
Show a benchmark run's configuration, agents, progress, result totals, and metrics.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs info
```
| Argument | Type | Description |
| ------------ | ---------------- | ------------------------ |
| `name_or_id` | `str` (required) | Benchmark run name or ID |
The summary reports status, progress, duration, best pass\@1, and submission details. The Agents section scores each agent the way the run's Agent Results table does: rank, pass\@1 with its interval, the deepest pass\@k, and per-task cost, time, and tokens. Results totals report outcome counts, input and output tokens, and LLM Cost, which is model spend on your own provider keys and is not billed by Osmosis. The result includes `platform_url`, which opens the run in the Platform.
### benchmark runs logs
Show recent lifecycle logs for a benchmark run, oldest first. Use this to monitor progress and diagnose failures.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs logs [--limit N] [--cursor CURSOR]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ------- | ------------------------------------------- |
| `name_or_id` | `str` (required) | — | Benchmark run name or ID |
| `--limit` | `int` | `50` | Entries per page (1–200) |
| `--cursor` | `str` | — | Cursor from a previous page's `next_cursor` |
When older entries exist, `--json` output includes a non-null `next_cursor`. Pass it back with `--cursor` to page further back in time.
### benchmark runs stop
Stop a pending, queued, or running benchmark run.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs stop [--yes]
```
| Argument / Option | Type | Description |
| ----------------- | ---------------- | ---------------------------- |
| `name_or_id` | `str` (required) | Benchmark run name or ID |
| `-y`, `--yes` | flag | Skip the confirmation prompt |
### benchmark runs download
Download benchmark summary metrics, task-level results, result artifacts, or logs.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis benchmark runs download [--type TYPES] [--output PATH] [--overwrite] [--yes]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `name_or_id` | `str` (required) | — | Benchmark run name or ID |
| `--type` | `str` | `summary,results` | Comma-separated selector: `summary`, `results`, `artifacts`, `logs`, or `all`. Replaces the default selection. |
| `-o`, `--output` | `path` | `.osmosis/benchmarks//` | Run output root. Filenames and subdirectories below the root are fixed. |
| `--overwrite` | flag | — | Re-download files that already exist locally with a matching size. |
| `-y`, `--yes` | flag | — | Skip the confirmation prompt for large downloads. |
Downloads use a fixed run-scoped layout:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
.osmosis/benchmarks//
├── summary.csv
├── results.csv
├── logs.txt
└── artifacts//
```
The CLI skips complete local files unless `--overwrite` is set, writes partial downloads atomically, and reports any files that still fail after retrying. Re-run the same command to fetch only missing or incomplete files.
`pending` and `queued` runs do not have downloadable outputs. Downloads from a `running` run are snapshots; use `--overwrite` to refresh existing files.
***
## secret
Manage Platform `environment_secret` records. Train and eval configs reference them through [`[secrets].required`](/cli/config-files#env-and-secrets). Benchmark configs reference model (`api_key_secret`), harness (`harness_api_key_secret`), and judge (`judge_api_key_secret`) records, while some benchmarks require an implicit record such as HLE's `HF_TOKEN`. The platform never returns secret values — these commands show or accept names and metadata only. Values are read from a hidden interactive prompt or a named environment variable, never as a plaintext command-line argument.
Each secret has a **scope**:
* **Workspace** secrets are shared across the workspace. Creating, updating, or deleting them requires an admin or owner role.
* **Personal** secrets are private to you. When a workspace and a personal secret share a name, your personal value wins at run time. Use personal secrets for overrides, such as swapping in your own provider API key without affecting teammates.
Secret names must match `^[A-Z][A-Z0-9_]*$` (SCREAMING\_SNAKE\_CASE).
### secret list
List secrets visible to you in the current workspace (names and metadata only — never values).
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret list [--scope all|workspace|personal] [--limit N] [--all]
```
| Option | Type | Default | Description |
| --------- | ----- | ------- | -------------------------------------------------- |
| `--scope` | `str` | `all` | Filter by scope: `all`, `workspace`, or `personal` |
| `--limit` | `int` | `50` | Maximum number of secrets to show |
| `--all` | flag | — | Show all secrets |
The output includes a Scope column labeled **Workspace** or **Personal**.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret list
osmosis secret list --scope personal
```
### secret set
Create or update (upsert) a secret. The CLI reads the value from the env var named by `--env VARNAME`; without that flag, you type the value at a hidden interactive prompt. In `--json` or `--plain` (non-interactive) modes, you must pass `--env`.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set [--scope workspace|personal] [--env VARNAME]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ---------- | ---------------------------------------------------------------------- |
| `name` | `str` (required) | — | Secret name in `^[A-Z][A-Z0-9_]*$` |
| `--scope` | `str` | `personal` | `personal` (private to you) or `workspace` (shared with the workspace) |
| `--env` | `str` | — | Read the value from this environment variable |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Create or update your personal secret (default scope):
osmosis secret set OPENAI_API_KEY
# Read the value from an env var (recommended for scripts and CI):
OPENAI_API_KEY=sk-... osmosis secret set OPENAI_API_KEY --env OPENAI_API_KEY
# Workspace-shared secret (requires admin or owner role):
OPENAI_API_KEY=sk-... osmosis secret set OPENAI_API_KEY --scope workspace --env OPENAI_API_KEY
```
### secret delete
Delete a secret within the given scope. The CLI prompts you to confirm unless you pass `--yes`.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret delete [--scope workspace|personal] [--yes]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ---------- | ----------------------------- |
| `name` | `str` (required) | — | Secret name to delete |
| `--scope` | `str` | `personal` | Scope of the secret to delete |
| `-y`, `--yes` | flag | — | Skip the confirmation prompt |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret delete OPENAI_API_KEY --yes
osmosis secret delete OPENAI_API_KEY --scope workspace --yes
```
***
## dataset
Manage platform datasets for the current workspace repository.
| Command | Description |
| ----------------------------------------------------------- | ------------------------------------------------ |
| `osmosis dataset upload [--overwrite] [--yes]` | Upload a CSV, JSONL, or Parquet dataset |
| `osmosis dataset download [-o PATH] [--overwrite]` | Download a dataset file |
| `osmosis dataset list [--limit N] [--all]` | List datasets |
| `osmosis dataset info ` | Show dataset details and processing status |
| `osmosis dataset preview [--rows N]` | Preview uploaded dataset rows |
| `osmosis dataset validate ` | Validate a dataset locally |
| `osmosis dataset logs [--limit N] [--cursor CURSOR]` | Show lifecycle logs for a dataset (oldest first) |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset upload data/train.jsonl
osmosis dataset info train
osmosis dataset preview train --rows 5
```
Dataset names are derived from the file name without its extension. If a dataset with that name already exists, the upload fails — pass `--overwrite` to replace it (the old record is soft-deleted).
Use `osmosis dataset logs ` to diagnose failed uploads. `--limit` accepts 1–200 entries (default: 50). When older entries exist, the `--json` output includes a non-null `next_cursor`; pass it back with `--cursor` to page further back in time.
`osmosis dataset upload` requires `--yes` (`-y`) in non-interactive modes (`--json`, `--plain`, or when stdin is piped). Running without it raises `INTERACTIVE_REQUIRED`, even with `--overwrite`. Add `--yes` to CI jobs and scripted uploads to skip the confirmation prompt.
***
## train
Submit and manage training runs for the current workspace repository.
### train submit
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train submit [--yes]
```
| Argument / Option | Type | Description |
| ----------------- | ----------------- | --------------------------------------- |
| `config_path` | `path` (required) | Training TOML under `configs/training/` |
| `-y`, `--yes` | flag | Skip confirmation prompt |
Config values come from the local TOML file. Training code comes from the synced workspace repository.
### train info
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train info [--output PATH]
```
Shows run details, checkpoints, and metrics. The summary panel includes progress (`current_step` / `total_steps` with percent complete) and the latest reward while the run is in flight. In rich mode, metrics are saved under `.osmosis/metrics/` by default.
### train logs
Show the most recent lifecycle logs for a training run, oldest first. Use this to diagnose failed or crashed runs.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train logs [--limit N] [--cursor CURSOR]
```
| Argument / Option | Type | Default | Description |
| ----------------- | ---------------- | ------- | ------------------------------------------- |
| `name` | `str` (required) | — | Training run name |
| `--limit` | `int` | `50` | Entries per page (1–200) |
| `--cursor` | `str` | — | Cursor from a previous page's `next_cursor` |
When older entries exist, `--json` output includes a non-null `next_cursor`. Pass it back with `--cursor` to page further back in time.
### Other train commands
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train list [--limit N] [--all]
osmosis train stop [--yes]
```
`train list` shows status, current step / total steps, and the latest reward for each run.
***
## model
Manage base (foundation) models and LoRA models produced by training runs. Deploying a LoRA model exposes it for inference.
### model list
List base models and LoRA models for the current workspace as two independently paginated sections (base first, then LoRA). The base table shows Name, Created, and Created By. The LoRA table shows Name, Base Model, Training Run, Checkpoint Step, Training Reward, and Created. When deployment info is available, the LoRA table also shows Deployment Status and a deployment-quota summary below it (for example, `2 of 5 inference deployments used`).
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list [--type all|base|lora] [--limit N] [--all]
```
| Option | Type | Default | Description |
| --------- | ----- | ------- | ------------------------------------------------- |
| `--type` | `str` | `all` | Filter to a single list: `all`, `base`, or `lora` |
| `--limit` | `int` | `50` | Maximum number of models to show per type |
| `--all` | flag | — | Show all models of each type |
`--limit` and `--all` apply to each list independently, and each list carries its own pagination cursor (`next_offset`).
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list
osmosis model list --type lora
osmosis model list --type base --limit 50
osmosis --json model list
```
`--json` output keys each list separately, so the structure itself identifies which list is which:
```json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{
"base_models": { "items": [...], "next_offset": null },
"lora_models": { "items": [...], "next_offset": null },
"active_deployments": 2,
"max_active_deployments": 5
}
```
When deployment info is available, the `active_deployments` / `max_active_deployments` quota keys appear for `--type all` and `--type lora`, but not for `--type base`.
### model info
Show details for a single LoRA model: base model, training run, checkpoint step, training reward, Hugging Face export status, and deployment status when deployment info is available.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model info
```
### model deploy
Deploy or reactivate a LoRA model by name.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model deploy
```
### model undeploy
Transition a LoRA model's deployment to inactive (idempotent). The LoRA model remains in the training run history.
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model undeploy
```
The standalone `osmosis deployment`, `osmosis deploy`, and `osmosis undeploy` commands have been removed. Use `osmosis model deploy ` and `osmosis model undeploy ` instead.
***
## upgrade
Self-upgrade the CLI to the latest version published on PyPI.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis upgrade
```
The CLI auto-detects your install method (`pip`, `pipx`, or `uv tool`) and runs the appropriate upgrade command.
# Configuration Files
Source: https://docs.osmosis.ai/cli/config-files
Reference TOML configuration files used by the Osmosis CLI
The Osmosis CLI uses TOML files for evaluation runs, training runs, and benchmark runs. Configs must live inside the workspace directory:
| Config type | Required location | Command |
| ----------- | -------------------------- | -------------------------- |
| Eval | `configs/eval/*.toml` | `osmosis eval submit` |
| Training | `configs/training/*.toml` | `osmosis train submit` |
| Benchmark | `configs/benchmark/*.toml` | `osmosis benchmark submit` |
Required fields are shown un-commented. Optional fields are commented out in template files and can be omitted to use platform defaults.
***
## Eval Config
Used by [`osmosis eval submit`](/cli/command-reference#eval-submit) to submit an evaluation run. The platform clones the workspace repository identified by the `origin` remote and runs the rollout server-side against a platform dataset.
```toml configs/eval/my-rollout.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
rollout = "my-rollout" # Rollout directory under rollouts/
entrypoint = "main.py" # Entrypoint relative to rollout dir
model_path = "openai/gpt-5-mini" # LiteLLM-style model name for the evaluation policy
dataset = "my-platform-dataset" # Platform dataset name from `osmosis dataset list`
# branch = "my-feature" # Use a pushed branch (default branch if omitted)
# commit_sha = "abc123..." # Pin code to a specific commit
[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200 # First N rows; omit for random 10% sample
# n = 1 # Evaluation attempts per row
# batch_size = 1 # Rows evaluated per batch
# pass_threshold = 1.0 # Minimum passing score
# agent_workflow_timeout_s = 450 # Agent workflow timeout per row
# grader_timeout_s = 150 # Grader timeout per row
# [env]
# LOG_LEVEL = "INFO" # Non-secret literal env var
[secrets]
# Required for eval configs. Default OpenAI eval models need this.
# Use required = [] when the evaluation needs no secret refs.
required = ["OPENAI_API_KEY"]
```
### `[experiment]`
| Field | Type | Required | Description |
| ------------ | ----- | -------- | --------------------------------------------------------------------------------- |
| `rollout` | `str` | Yes | Rollout directory name under `rollouts/` |
| `entrypoint` | `str` | Yes | Python entrypoint relative to the rollout directory |
| `model_path` | `str` | Yes | LiteLLM-style model name for the evaluation policy (e.g. `openai/gpt-5-mini`) |
| `dataset` | `str` | Yes | Platform dataset name from `osmosis dataset list` |
| `branch` | `str` | No | Use the current head of this pushed branch. Mutually exclusive with `commit_sha`. |
| `commit_sha` | `str` | No | Pin code to a specific commit. Mutually exclusive with `branch`. |
Omit both `branch` and `commit_sha` to use the latest synced commit on the repository's default branch. Branch submissions resolve the branch head once, so the run remains pinned to the resulting full commit SHA.
### `[evaluation]`
All fields are optional. Omit values to use platform defaults.
| Field | Type | Description |
| -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------- |
| `limit` | `int` | Number of rows to evaluate (the first `N` rows). When omitted, the platform evaluates a random 10% sample of the dataset. |
| `n` | `int` | Number of evaluation attempts per row (use values > 1 for pass\@n metrics) |
| `batch_size` | `int` | Rows evaluated per batch |
| `pass_threshold` | `float` | Score at or above which a sample counts as passing |
| `agent_workflow_timeout_s` | `float` | Timeout for `AgentWorkflow.run()` per row |
| `grader_timeout_s` | `float` | Timeout for `Grader.grade()` per row |
### `[env]` and `[secrets]` (evaluation)
Optional `[env]` variables and a required `[secrets]` table for the evaluation run container. Eval configs must include `[secrets]` — use `required = []` only when the evaluation needs no secret refs. See [`[env]` and `[secrets]`](#env-and-secrets) below for the full ruleset.
***
## Benchmark Config
Used by [`osmosis benchmark submit`](/cli/command-reference#benchmark-submit) to run a benchmark already added to the current workspace. Benchmark configs describe the task selection, agent harnesses and models, and execution settings. They do not reference workspace rollout code.
```toml configs/benchmark/terminal-bench-smoke.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
benchmark = "terminal-bench-2-1" # Benchmark key from `benchmark list`
# Omit [tasks] to run all tasks.
[tasks]
task_names = ["terminal-bench/git-multibranch"] # Explicit task names
# categories = ["software-engineering"] # Union tasks from these categories
# task_set = "parity" # Benchmark-defined parity sample
[[agents]]
harness = "codex"
[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY" # Platform secret record name
# [agents.env]
# AGENT_MODE = "strict" # Literal env for this agent
[execution]
attempts_per_task = 1
max_concurrent_attempts = 4
timeout_multiplier = 1.0
max_retries = 0
# pass_threshold = 1.0
# Judge-enabled benchmarks only:
# judge_model = "openai/gpt-5.2"
# judge_api_key_secret = "OPENAI_API_KEY"
# [env]
# LOG_LEVEL = "info" # Literal env for every agent
```
### `[experiment]` (benchmark)
| Field | Type | Required | Description |
| ----------- | ----- | -------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `benchmark` | `str` | Yes | A benchmark already added to the current workspace, given as its key, name, or ID. All three are exact, case-sensitive matches |
[`benchmark list`](/cli/command-reference#benchmark-list) shows both the key and the name; either one works here, as does the ID.
### `[tasks]`
All fields are optional. Omit the section to run every task. `task_names` and `categories` are unioned when both are set. When `task_set` is set, its published sample is used instead of `task_names` or `categories`.
| Field | Type | Description |
| ------------ | ----------- | ----------------------------------------------------------- |
| `task_names` | `list[str]` | Explicit task names from the benchmark manifest |
| `categories` | `list[str]` | Include every task in the named categories |
| `task_set` | `str` | Named sample published by the benchmark; currently `parity` |
For HLE, we recommend `task_set = "parity"`. Omit `[tasks]` only when you intend to submit the full HLE benchmark.
### `[[agents]]` and `[agents.model]`
Each run requires one to eight agents. Harness availability depends on the selected benchmark. Across managed benchmarks, supported `harness` values are `claude-code`, `codex`, `cursor-cli`, `gemini-cli`, `mini-swe-agent`, `openhands`, `opencode`, and `terminus-2`. The Platform rejects a harness that is unavailable for the selected benchmark. Every agent needs its own `[[agents]]` entry regardless: a benchmark that runs only its official scaffold rejects every harness, and one that merely allows a harness runs its official scaffold when you omit the field. `benchmark info` reports which of the three applies.
| Agent field | Required | Description |
| ------------------------ | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `harness` | Benchmark-dependent | Agent harness offered by the selected benchmark; omit to run the benchmark's official scaffold, where it has one |
| `harness_api_key_secret` | `cursor-cli` and `mini-swe-agent` only | `CURSOR_API_KEY` or `MSWEA_API_KEY` respectively, holding that harness's credential; omit for every other harness |
| `model` | Yes | Provider, endpoint, or hosted model configuration |
| `env` | No | Literal environment variables for this agent |
| Model type | Required model fields | Use for |
| ---------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `provider` | `model`, `api_key_secret` | Provider models such as `openai/gpt-5.2` |
| `endpoint` | `base_url`, `model`, `api_key_secret` | OpenAI-compatible custom endpoints; `extra_headers` is optional |
| `hosted` | `base_model`, `lora_model_name` | A LoRA model you trained on Osmosis and deployed. `base_model` is the base it was trained on; `lora_model_name` is the LoRA model name |
For provider and endpoint models, `api_key_secret` is the model's Platform secret record name. `harness_api_key_secret` is separate and per-agent: it is required for `cursor-cli` and `mini-swe-agent`, and rejected for every other harness. Set it to `CURSOR_API_KEY` for `cursor-cli` and to `MSWEA_API_KEY` for `mini-swe-agent`. Those are the variables the harnesses read, and any other value is rejected at submit.
Register the record with [`osmosis secret set`](/cli/command-reference#secret-set), then reference its name in the agent. For example, a Cursor CLI agent can use:
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[[agents]]
harness = "cursor-cli"
harness_api_key_secret = "CURSOR_API_KEY"
[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis secret set CURSOR_API_KEY
```
To benchmark one of your own LoRA models instead of a provider model, use `type = "hosted"`. Take both values from `osmosis model list --type lora`: `base_model` is the LoRA's Base Model column and `lora_model_name` is its Name column.
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[[agents]]
harness = "codex"
[agents.model]
type = "hosted"
base_model = "Qwen/Qwen3-8B"
lora_model_name = "deep-swe-agent"
```
Deploy the LoRA model with [`osmosis model deploy`](/cli/command-reference#model-deploy) before submitting; a run against an undeployed one is rejected, as is a `base_model` that does not match what that LoRA model was trained on. Hosted agents need no `api_key_secret`, since Osmosis serves them.
Each agent's effective environment combines top-level `[env]` with that agent's `[agents.env]`. A provider or endpoint agent's `api_key_secret` name cannot appear in that agent's effective environment. The `judge_api_key_secret` name cannot appear in top-level `[env]` or any agent's `[agents.env]`.
For a `cursor-cli` agent, do not also set `CURSOR_API_KEY` in either env table; for a `mini-swe-agent` agent, do not set `MSWEA_API_KEY`. The resolved harness secret owns that destination variable.
Provider and endpoint `api_key_secret` fields also cannot reference runner-reserved names: `HF_TOKEN`, `DAYTONA_API_KEY`, `DAYTONA_API_URL`, `SKYPILOT_SERVICE_ACCOUNT_TOKEN`, or `SKYPILOT_API_SERVER_ENDPOINT`. `HF_TOKEN` is valid only as HLE's implicit Platform secret record; it cannot be a model secret reference or be defined as an environment-variable key in any benchmark `[env]` or `[agents.env]`.
All secret fields contain record names, never credential values. `[agents.env]` contains literal variables for one agent; top-level `[env]` applies to every agent, and agent-specific values override the same global key.
### `[execution]`
| Field | Type | Default | Range | Description |
| ------------------------- | ------- | ----------------- | ----------------- | ------------------------------------------------------- |
| `attempts_per_task` | `int` | `1` | `1–10` | Attempts run for each selected task |
| `max_concurrent_attempts` | `int` | `4` | `1–64` | Maximum attempts running concurrently |
| `timeout_multiplier` | `float` | `1.0` | `> 0` and `<= 10` | Multiplier applied to benchmark task timeouts |
| `max_retries` | `int` | `0` | `0–5` | Retries for retryable infrastructure failures |
| `pass_threshold` | `float` | benchmark default | `0–1` | Score at or above which an attempt passes |
| `judge_model` | `str` | benchmark default | — | Optional judge model override for HLE and GDPVal |
| `judge_api_key_secret` | `str` | — | — | Required Platform secret record name for HLE and GDPVal |
HLE and GDPVal require `judge_api_key_secret`. Register that record with [`osmosis secret set`](/cli/command-reference#secret-set). `judge_model` is optional for those benchmarks and uses the benchmark default when omitted. Benchmarks without an LLM judge reject both `judge_model` and `judge_api_key_secret`.
Never put secret values in `[env]`, `[agents.env]`, or `extra_headers`. Benchmark configs use Osmosis fields and are not Harbor configuration files.
***
## Training Config
Used by [`osmosis train submit`](/cli/command-reference#train-submit) to submit a training run.
```toml configs/training/my-rollout.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
rollout = "my-rollout" # Rollout directory under rollouts/
entrypoint = "main.py" # Entrypoint file name
model_path = "Qwen/Qwen3.6-35B-A3B" # Supported base model
dataset = "my-dataset" # Platform dataset name
# branch = "my-feature" # Use a pushed branch (default branch if omitted)
# commit_sha = "abc123..." # Pin code to a commit
[training]
# lr = 1e-6 # Learning rate
# total_epochs = 1 # Training epochs
# n_samples_per_prompt = 8 # Rollout samples per prompt
# rollout_batch_size = 32 # Rollout batch size
# max_prompt_length = 8192 # Max prompt tokens
# max_response_length = 8192 # Max response tokens
# agent_workflow_timeout_s = 450 # Agent timeout per row
# grader_timeout_s = 150 # Grader timeout per row
[sampling]
# rollout_temperature = 1.0 # Sampling temperature
# rollout_top_p = 1.0 # Top-p sampling
[checkpoints]
# eval_interval = 10 # Evaluate every N rollouts
# checkpoint_save_freq = 20 # Save checkpoint every N rollouts
# [advanced]
# Backend-specific fields. Use only when instructed by Osmosis support.
# [env]
# LOG_LEVEL = "INFO" # Non-secret literal env var
# [secrets]
# required = ["OPENAI_API_KEY"] # Optional in training; if set, must include `required`
```
Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set `branch` to use a pushed branch or `commit_sha` for a specific pushed revision; omit both to use the default branch.
### `[experiment]`
| Field | Type | Required | Description |
| ------------ | ----- | -------- | ---------------------------------------------------------------------------------------- |
| `rollout` | `str` | Yes | Rollout directory name under `rollouts/` |
| `entrypoint` | `str` | Yes | Python entrypoint file name, usually `main.py` |
| `model_path` | `str` | Yes | Supported base model path |
| `dataset` | `str` | Yes | Dataset name from `osmosis dataset list` |
| `branch` | `str` | No | Use the current head of this pushed branch. Mutually exclusive with `commit_sha`. |
| `commit_sha` | `str` | No | Git commit SHA to fetch from the workspace repository. Mutually exclusive with `branch`. |
Omit both `branch` and `commit_sha` to use the latest synced commit on the repository's default branch. Branch submissions resolve the branch head once, so the run remains pinned to the resulting full commit SHA.
### `[training]`
| Field | Type | Default | Description |
| -------------------------- | ------- | ---------------- | ------------------------------------ |
| `lr` | `float` | platform default | Learning rate |
| `total_epochs` | `int` | platform default | Number of training epochs |
| `n_samples_per_prompt` | `int` | platform default | Rollout samples generated per prompt |
| `rollout_batch_size` | `int` | platform default | Prompts processed per rollout batch |
| `max_prompt_length` | `int` | platform default | Maximum prompt tokens |
| `max_response_length` | `int` | platform default | Maximum response tokens |
| `agent_workflow_timeout_s` | number | platform default | Agent rollout timeout per row |
| `grader_timeout_s` | number | platform default | Grader timeout per row |
### `[sampling]`
| Field | Type | Default | Description |
| --------------------- | ------ | ---------------- | ------------------------------------ |
| `rollout_temperature` | number | platform default | Sampling temperature during rollouts |
| `rollout_top_p` | number | platform default | Top-p sampling threshold |
### `[checkpoints]`
| Field | Type | Default | Description |
| ---------------------- | ----- | ---------------- | -------------------------------------------- |
| `eval_interval` | `int` | platform default | Evaluate every N rollout steps |
| `checkpoint_save_freq` | `int` | platform default | Save a LoRA checkpoint every N rollout steps |
### `[advanced]`
Optional backend-specific fields. The CLI preserves unknown keys in this section and the platform validates them server-side.
### `[env]` and `[secrets]`
Use these sections to inject environment variables into the rollout container during training runs or evaluation runs. The same shape applies to both training and evaluation configs.
| Section | Values | Use for |
| -------------------- | -------------------------------------------------- | -------------------------------- |
| `[env]` | Literal strings stored in the config file | Non-secret configuration |
| `[secrets].required` | List of platform `environment_secret` record names | API keys and private credentials |
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[env]
LOG_LEVEL = "INFO"
[secrets]
required = ["OPENAI_API_KEY", "DATABASE_URL"]
```
Rules:
* `[env]` keys must match `^[A-Z_][A-Z0-9_]*$`; `[secrets].required` names must match `^[A-Z][A-Z0-9_]*$`.
* The same name cannot appear in both `[env]` and `[secrets].required`.
* `[env]` keys starting with `_OSMOSIS_` are reserved by the platform and cannot be used.
* `[secrets].required` entries are record names only. The platform resolves each name to its encrypted value server-side and injects it as an env var of the same name. Secret values never appear in the config file, the API payload, or CLI output.
* Eval configs must include `[secrets]`. Use `required = []` only when the evaluation needs no secret refs.
* Training configs may omit `[secrets]`. If you include the table, it must define `required`.
Secrets are scoped. A **workspace** secret is shared across the workspace; a **personal** secret is private to you and overrides the workspace secret of the same name at run time. Register secrets with [`osmosis secret set`](/cli/command-reference#secret) before submitting a run that references them.
Start with only `[experiment]` (plus `[secrets]` for eval configs) and let the platform use training defaults. Add optional fields only when you need to tune a run.
# Installation & Authentication
Source: https://docs.osmosis.ai/cli/installation
Install the Osmosis CLI and authenticate with the platform
## Installation
Install the Osmosis CLI from PyPI:
```bash pip theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install osmosis-ai
```
```bash pipx theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pipx install osmosis-ai
```
```bash uv tool theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
uv tool install osmosis-ai
```
Python **3.12 or later** is required. The package registers three equivalent CLI aliases: `osmosis`, `osmosis-ai`, and `osmosis_ai`.
Verify the installation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis --version
```
### SDK feature extras
The base `osmosis-ai` distribution contains the CLI and framework-neutral
rollout core. A rollout entrypoint normally combines `server` with its agent
integration or backend; CLI-only workflows add `rubric` or `parquet` only when
those commands need them.
| Extra | Adds | Used by |
| --------------- | ----------------------------------------------------------- | --------------------------------------- |
| `server` | FastAPI rollout server | Rollout entrypoints |
| `strands` | Strands Agents integration | Strands rollouts |
| `openai-agents` | OpenAI Agents SDK integration | OpenAI Agents rollouts |
| `harbor` | Harbor execution backend without a bundled SkyPilot runtime | Harbor rollouts and SDK harnesses |
| `rubric` | LLM-as-judge rubric evaluation | `osmosis eval rubric` |
| `parquet` | Parquet dataset validation | Dataset upload and validation |
| `full` | All optional features | Development or all-feature environments |
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install "osmosis-ai[server,strands]>=0.3,<0.4"
```
Do not install Harbor's `skypilot` extra in a managed rollout. Use
`osmosis-ai[harbor]`; the rollout runtime supplies the compatible SkyPilot
SDK.
### Upgrading
The CLI can upgrade itself in-place. It auto-detects your install method (`pip`, `pipx`, or `uv tool`) and runs the appropriate upgrade command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis upgrade
```
You can also upgrade manually with your package manager of choice:
```bash pip theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade osmosis-ai
```
```bash pipx theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pipx upgrade osmosis-ai
```
```bash uv tool theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
uv tool upgrade osmosis-ai
```
### Version compatibility
The CLI sends its version to the platform on every request so the platform can signal when an upgrade is recommended or required:
* **Deprecation warning.** If your installed version is approaching end of support, the CLI prints a one-time yellow `⚠` warning to stderr. Commands continue to run normally — schedule an upgrade at your convenience.
* **Upgrade required.** If your version is below the minimum supported version, platform requests fail with an `Upgrade required` error and the command exits. Run `osmosis upgrade` (or the equivalent for your install method) to continue.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis upgrade
```
## Authentication
### Login
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis auth login
```
Opens a browser-based OAuth flow to authenticate your CLI session. Credentials are stored locally and reused for subsequent commands.
| Option | Description |
| ----------------- | ------------------------------------------------------------------------------------ |
| `-f`, `--force` | Force re-login, clearing existing credentials |
| `--token ` | Authenticate with a personal access token instead of browser flow (useful for CI/CD) |
For CI/CD pipelines, set the `OSMOSIS_TOKEN` environment variable instead of running `osmosis auth login`. When this variable is set, the CLI uses it automatically and the `login` command is disabled.
### Logout
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis auth logout
```
Revokes the current session and clears local credentials.
| Option | Description |
| ------------- | ------------------------ |
| `-y`, `--yes` | Skip confirmation prompt |
### Who Am I
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis auth whoami
```
Displays the currently authenticated user and token expiration date.
## Workspace Context
Most platform commands are scoped through the workspace repository you are currently inside. If you are setting up Osmosis for the first time, complete [Onboarding](/platform/onboarding) before running workspace-scoped commands.
For details on how the CLI reads GitHub `origin` and maps local commands to a platform workspace, see [Workspace Repository](/cli/workspace/repository).
## Next Steps
Set up a workspace repository and local CLI context.
Full reference for every CLI command and its options.
# Building AgentWorkflows
Source: https://docs.osmosis.ai/cli/rollout/agent-workflows
Implement the AgentWorkflow class to define your agent behavior for training
`AgentWorkflow` is the SDK contract for rollout behavior. You subclass it, implement one async `run()` method, and create one sample by calling the current policy through an Osmosis-supported agent integration.
The workflow should answer one question: **given this dataset prompt, what should the agent do before the grader scores the result?**
## Base Class
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
class MyWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
# Build and run your agent here.
pass
```
The SDK shape is:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class AgentWorkflow(Generic[TConfig], ABC):
def __init__(self, config: TConfig | None = None):
self.config = config
@abstractmethod
async def run(self, ctx: AgentWorkflowContext[TConfig]) -> Any:
raise NotImplementedError
```
`run()` is called once for each workflow execution. It should construct any per-execution agent/session objects inside the method, run the agent, and let the integration register the resulting conversation with the active `RolloutContext`.
## AgentWorkflowContext
The `ctx` object gives the workflow its input and config:
| Field | Type | Description |
| ------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ctx.prompt` | `list[dict[str, Any]]` | Input messages for the current dataset row |
| `ctx.config` | `TConfig \| None` | Custom workflow config object, if one was provided |
| `ctx.metadata` | `dict[str, Any] \| None` | Per-row metadata from the dataset's optional `metadata` column. `None` when the row has no metadata. |
| `ctx.artifacts_dir` | `pathlib.Path \| None` | Per-rollout directory where you can write log, trace, and other output files. `None` when the execution environment can't provision a writable directory, so check for `None` before writing files. |
If your dataset row contains `system_prompt`, `user_prompt`, and `ground_truth`, the prompt fields are assembled into `ctx.prompt`. The reference answer is not passed to the workflow; it is exposed to your grader as `ctx.label`. The same `metadata` object is available on both `AgentWorkflowContext` and `GraderContext`, so workflows and graders can read the same per-row context.
Keep task answers out of `AgentWorkflow.run()`. The workflow should produce behavior; the `Grader` should decide whether that behavior deserves reward.
### Writing Artifacts
Use `ctx.artifacts_dir` to write files that shouldn't be embedded in the sample payload — logs, traces, screenshots, or other large or binary outputs. Each rollout gets its own directory, but it's `None` when the environment can't provision one, so guard with `if ctx.artifacts_dir:` before writing to it — an unguarded write raises and fails the workflow.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
class TracingWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
if ctx.artifacts_dir:
(ctx.artifacts_dir / "run.log").write_text("started\n")
...
```
After the rollout finishes, collected files appear alongside its sample in the run's **Artifacts** panel on the Osmosis Platform, mirroring the layout you write under `ctx.artifacts_dir`. Artifact collection never affects rewards or rollout status.
### Saved Trajectories
The SDK saves every finished rollout as an [ATIF](https://www.harborframework.com/docs/agents/trajectory-format) (Agent Trajectory Interchange Format) document alongside the run's artifacts. Saving is automatic — you don't opt in or configure anything — and it never affects rewards or rollout status.
Files land next to the artifacts directory on the platform-managed host:
```
~/.osmosis//
├── trajectory.json # ATIF document for the rollout's sample
└── artifacts/... # files you wrote under ctx.artifacts_dir
```
The saved transcript is a normalized, chat-completions-shaped view of the conversation. The supported integrations (`OsmosisStrandsAgent`, `OsmosisAgent`) produce it for you, so you don't need to change your workflow. Your grader and any callbacks still see the framework-native `messages` — the SDK uses the normalized view only for persistence.
If you build a custom sample source whose native history isn't already OpenAI chat-completions-shaped, set `RolloutSample.trajectory_messages` on the returned sample to control what gets persisted (an explicit `None` skips trajectory saving for that sample).
Assistant messages produced through the integrations already carry per-call `usage`, `model`, and timestamp metadata, which the SDK maps into ATIF `Step.metrics`, `Step.model_name`, and `Step.timestamp`. Custom workflows that manage their own message list can populate the same slots by copying `response.usage` / `response.model` / `response.created_at` onto each assistant message before appending it.
## Model Routing Requirement
LLM calls inside `run()` **must** route through the `RolloutContext` installed by the execution backend. The training cluster uses the rollout-scoped chat-completions URL from this context to serve the current policy, collect traces, and connect the reward to the sample.
Use one of the supported integrations:
| Framework | Use | Integration objects |
| ----------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------- |
| Strands Agents | Strands tools, Strands message history, migration from `strands.Agent` | `OsmosisStrandsAgent`, `OsmosisRolloutModel` |
| OpenAI Agents SDK | `Runner.run`, sessions, handoffs, OpenAI-style tools | `OsmosisAgent`, `OsmosisRolloutModel`, `OsmosisMemorySession` |
Do not call `litellm`, the OpenAI SDK, or another provider SDK directly with a hard-coded policy model from `run()`. Direct calls bypass the rollout context and are not compatible with training.
## Strands Pattern
For Strands, pass `ctx.prompt` directly as `messages` and call `invoke_async()`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
from osmosis_ai.rollout.integrations.agents.strands import (
OsmosisRolloutModel,
OsmosisStrandsAgent,
)
class SimpleStrandsWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisStrandsAgent(
name="simple-strands-agent",
model=OsmosisRolloutModel(params={"temperature": 1.0}),
messages=ctx.prompt,
callback_handler=None,
)
await agent.invoke_async()
```
Constructing `OsmosisStrandsAgent` inside `run()` binds it to the active rollout context and registers the agent as a sample source.
See [Strands Integration](/cli/rollout/strands-integration) for tool examples, migration steps, and details about `OsmosisRolloutModel`.
## OpenAI Agents Pattern
For OpenAI Agents, construct an `OsmosisAgent`, attach OpenAI Agents `ModelSettings`, create one `OsmosisMemorySession`, and pass that session to `Runner.run()`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from agents import ModelSettings, Runner
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
from osmosis_ai.rollout.integrations.agents.openai_agents import (
OsmosisAgent,
OsmosisMemorySession,
OsmosisRolloutModel,
)
class SimpleOpenAIWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisAgent(
name="simple-openai-agent",
instructions="Answer the user's request clearly.",
model=OsmosisRolloutModel(),
model_settings=ModelSettings(temperature=1.0, max_tokens=4096),
)
session = OsmosisMemorySession()
await Runner.run(
agent,
ctx.prompt,
session=session,
)
```
The session is what records the OpenAI Agents SDK conversation for grading. Create it inside `run()` so it registers with the current `RolloutContext`.
See [OpenAI Agents Integration](/cli/rollout/openai-agents-integration) for session behavior, tracing notes, and migration steps.
## Custom Configuration
Custom configs extend `AgentWorkflowConfig`. Define a module-level config instance in your rollout entrypoint and pass it to the backend:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import (
AgentWorkflow,
AgentWorkflowConfig,
AgentWorkflowContext,
ConcurrencyConfig,
)
class SearchWorkflowConfig(AgentWorkflowConfig):
name: str = "search-workflow"
max_iterations: int = 8
temperature: float = 1.0
concurrency: ConcurrencyConfig = ConcurrencyConfig(max_concurrent=4)
class SearchWorkflow(AgentWorkflow[SearchWorkflowConfig]):
async def run(self, ctx: AgentWorkflowContext[SearchWorkflowConfig]) -> None:
config = ctx.config or SearchWorkflowConfig()
max_iterations = config.max_iterations
temperature = config.temperature
# Use these values when constructing your agent.
search_workflow_config = SearchWorkflowConfig()
```
`osmosis train submit` preflight auto-discovers at most one module-level `AgentWorkflowConfig` instance from the entrypoint module. Eval and training TOML files do not currently set workflow config fields directly.
`BaseConfig` allows extra fields, so simple rollout configs usually do not need additional Pydantic boilerplate.
| Field | Type | Default | Description |
| ------------- | ------------------- | --------- | -------------------------------------- |
| `name` | `str` | required | Identifier for the workflow |
| `description` | `str \| None` | `None` | Optional description |
| `concurrency` | `ConcurrencyConfig` | unlimited | Maximum concurrent workflow executions |
## Tool-Using Workflows
Tool use belongs inside your agent framework, not in the backend. Define tools the way your framework expects, pass them into the Osmosis-wrapped agent, and let the integration record the resulting messages.
For example, a Strands workflow can keep its tool list in config:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from typing import Any
from strands import tool
from osmosis_ai.rollout import (
AgentWorkflow,
AgentWorkflowConfig,
AgentWorkflowContext,
)
from osmosis_ai.rollout.integrations.agents.strands import (
OsmosisRolloutModel,
OsmosisStrandsAgent,
)
@tool(name="search")
def search_tool(query: str) -> str:
"""Search for information."""
return f"results for {query}"
class ToolWorkflowConfig(AgentWorkflowConfig):
name: str = "tool-workflow"
model: Any = OsmosisRolloutModel(params={"temperature": 1.0})
tools: Any = [search_tool]
max_iterations: int = 8
tool_workflow_config = ToolWorkflowConfig()
class ToolWorkflow(AgentWorkflow[ToolWorkflowConfig]):
async def run(self, ctx: AgentWorkflowContext[ToolWorkflowConfig]) -> None:
config = ctx.config or ToolWorkflowConfig()
agent = OsmosisStrandsAgent(
name="search-agent",
model=config.model,
tools=config.tools,
messages=ctx.prompt,
callback_handler=None,
)
for _ in range(config.max_iterations):
result = await agent.invoke_async()
content = result.message.get("content", [])
if not any("toolUse" in block for block in content):
break
```
## Auto-Discovery
`osmosis train submit` and `osmosis eval submit` both scan the rollout entrypoint module for concrete `AgentWorkflow` subclasses. You do not need decorators or registration functions, but the entrypoint still needs to construct a backend and serve it with `create_rollout_server()` so the rollout server can run on the platform.
For training preflight, your entrypoint file must contain exactly **one** concrete `AgentWorkflow` subclass. If the SDK finds zero or more than one, `osmosis train submit` fails during discovery.
Use helpers, base classes, tools, and configs freely, but keep only the workflow class you want to run as a concrete `AgentWorkflow` in the entrypoint.
## Next Steps
Build a Strands-based rollout with tools and `OsmosisStrandsAgent`.
Build an OpenAI Agents SDK rollout with `OsmosisAgent` and `OsmosisMemorySession`.
Define reward logic for the sample your workflow produces.
Submit an evaluation run to test your workflow and grader before a training run.
# Evaluation
Source: https://docs.osmosis.ai/cli/rollout/eval
Submit evaluation runs and inspect results from your workspace directory
An evaluation run submits against a platform dataset using the same workspace, rollout, entrypoint, dataset, and optional `branch` or `commit_sha` semantics as [`osmosis train submit`](/cli/command-reference#train-submit). The two source fields are mutually exclusive. The platform clones the repository identified by the workspace directory's `origin` remote and executes the rollout server-side, so push your changes before submitting.
Evaluation configs must live under `configs/eval/` inside a structured Osmosis workspace directory.
`osmosis eval submit` is also the recommended pre-flight before a training run — run it first to catch problems before committing GPU time.
## Quick Start
From inside your workspace directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset list # confirm the platform dataset name
git push # make sure the platform sees your commit
osmosis eval submit configs/eval/my-rollout.toml
```
Then inspect or manage the run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval list
osmosis eval info
osmosis eval stop
```
## Evaluation Config
See [Config Files](/cli/config-files#eval-config) for the full field reference.
```toml configs/eval/my-rollout.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
rollout = "my-rollout" # Rollout directory under rollouts/
entrypoint = "main.py" # Entrypoint relative to the rollout directory
model_path = "openai/gpt-5-mini" # LiteLLM-style model name for the evaluation policy
dataset = "my-platform-dataset" # Platform dataset name from `osmosis dataset list`
# branch = "my-feature" # Optional: use a pushed branch (default branch if omitted)
# commit_sha = # Optional: pin to a specific commit
[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200
# n = 1
# batch_size = 1
# pass_threshold = 1.0
# agent_workflow_timeout_s = 450
# grader_timeout_s = 150
# [env]
# LOG_LEVEL = "INFO"
[secrets]
# Required for eval configs. Use required = [] only when no secrets are needed.
required = ["OPENAI_API_KEY"]
```
When `[evaluation].limit` is omitted, the platform evaluates a random 10% sample of the dataset (at least one row). Set `limit` to evaluate a fixed number of rows — the first `N` rows of the dataset, in order.
Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set `branch` to use a pushed branch or `commit_sha` for a specific pushed revision; omit both to use the default branch.
## How It Works
The CLI reads the evaluation TOML, resolves the workspace from the Git `origin` remote, and validates the `[experiment]` and `[secrets]` sections (plus optional `[evaluation]` and `[env]`) locally before submitting.
The CLI submits the evaluation run request. The platform resolves the selected `branch` or `commit_sha` once, clones that commit from the connected workspace repository, and prepares the evaluation environment.
Before evaluating any rows, the platform runs a pre-flight check that confirms `[experiment].model_path` is reachable with your configured credentials. If the model is unreachable — wrong name, missing or invalid API key, or provider rate limiting — the run fails early instead of consuming evaluation resources. Provide the model's provider API key by registering it with [`osmosis secret set`](/cli/command-reference#secret) and listing it under `[secrets].required` (see [Configuration Files](/cli/config-files#env-and-secrets)).
The platform starts your rollout, drives `AgentWorkflow.run(ctx)` for each selected row of the platform dataset using `[experiment].model_path` as the evaluation policy, then runs `Grader.grade(ctx)` against the row's `ground_truth`.
The platform aggregates rewards, pass rates, and per-row results. Use `osmosis eval info ` (or `osmosis --json eval info `) to inspect them.
## Commands
| Command | Description |
| ------------------------------------------- | ------------------------------------------------------------------------------- |
| `osmosis eval submit .toml [--yes]` | Submit an evaluation run from a TOML under `configs/eval/`. |
| `osmosis eval list [--limit N] [--all]` | List evaluation runs for the current workspace directory. |
| `osmosis eval info ` | Show details and results for a specific evaluation run. |
| `osmosis eval download ` | Download metrics, trajectories, artifacts, and logs for a run. |
| `osmosis eval stop [--yes]` | Stop a pending or running evaluation run. |
| `osmosis eval rubric` | Local LLM-as-judge over a JSONL conversation file. Does not touch the platform. |
See the [Command Reference](/cli/command-reference#eval) for the full flag list.
## From Evaluation Run to Training Run
Run `osmosis eval submit configs/eval/my-rollout.toml`. Use `osmosis eval list` and `osmosis eval info ` to track progress and inspect results.
Push fixes to the workspace repository and re-submit. Use `branch` for a feature branch or `commit_sha` to re-run against an older revision when comparing changes.
Once evaluation run results look healthy, run `osmosis train submit configs/training/my-rollout.toml`. See [Training Runs](/platform/training-runs).
## Download Run Outputs
Once a run has data, use `osmosis eval download` to pull metrics, trajectories, artifacts, and logs to your local disk instead of clicking through the web UI.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Default: metrics + trajectories for the whole run
osmosis eval download my-eval-run
# Only trajectories and artifacts for selected rows
osmosis eval download my-eval-run --type trajectories,artifacts --rows 3,7,10-20
# Everything, into a custom root
osmosis eval download my-eval-run --type all -o ./eval-outputs/
```
Files land under a fixed layout so re-running the command resumes cleanly:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
.osmosis/evals//
├── metrics.json
├── summary.jsonl
├── trajectories/row__run_.json
├── artifacts/row__run_/...
└── logs.txt
```
The CLI skips local files whose size matches the platform manifest unless you pass `--overwrite`, and downloads over 100 MiB require confirmation unless you pass `--yes`. The CLI retries failed files automatically and lists anything still missing so a second run picks it up. See the [Command Reference](/cli/command-reference#eval-download) for the full flag list.
`osmosis eval info -o` now points at the same run output root, and rich-mode metrics exports save to `.osmosis/evals//metrics.json`. Existing files under `.osmosis/metrics/` are left untouched.
## Local Rubric Scoring
`osmosis eval rubric` is a local utility for scoring an existing JSONL conversation file with an LLM judge. It does not require a workspace directory or platform authentication, and it does not run a rollout.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval rubric -d conversations.jsonl \
--rubric "Evaluate the assistant's helpfulness..." \
--model openai/gpt-5-mini
```
See the [Command Reference](/cli/command-reference#eval-rubric) for the full flag list.
## Next Steps
Full reference for evaluation and training configuration files.
Push and sync rollout code before submitting evaluation runs or training runs.
Submit a training run once evaluation run results look good.
# Execution Backends
Source: https://docs.osmosis.ai/cli/rollout/execution-backends
Choose between in-process and Harbor-managed rollout execution in the open source osmosis-ai SDK
Most users do not configure execution backends from the CLI. `osmosis eval submit` and `osmosis train submit` hand execution to the Osmosis platform, and the rollout entrypoint constructs the backend on the server side. This page is an SDK-level guide for users embedding the open source [`osmosis-ai`](https://pypi.org/project/osmosis-ai/) package in custom harnesses or self-hosted experiments.
An execution backend determines **where** an `AgentWorkflow` and `Grader` run when you orchestrate rollouts through the SDK directly.
| Backend | Runs where | Best for |
| --------------- | --------------------------------- | ------------------------------------------------------------- |
| `LocalBackend` | Current Python process | Fast local development, custom eval harnesses, debugging |
| `HarborBackend` | Harbor-managed trial environments | Per-trial isolation, task environments, dependency separation |
## Backend Responsibilities
Every backend has the same core responsibilities:
The request contains the input prompt for one dataset row and, if grading is enabled, its reference label and metadata.
The backend creates an `AgentWorkflowContext`, installs a `RolloutContext`, and calls `AgentWorkflow.run(ctx)`.
An agent integration such as `OsmosisStrandsAgent` or `OsmosisAgent` registers one sample source on the rollout context. The backend collects its sample after the workflow finishes.
If grading is available, the backend creates a `GraderContext` and calls `Grader.grade(ctx)`, or delegates verification to the task environment.
The backend returns an `ExecutionResult` with a status, one optional `sample`, and any categorized error.
## ExecutionBackend Interface
All backends implement this shape:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class ExecutionBackend(ABC):
async def execute(
self,
request: ExecutionRequest,
on_workflow_complete: ResultCallback,
on_grader_complete: ResultCallback | None = None,
) -> None: ...
@property
def max_concurrency(self) -> int: ... # 0 = no limit
@property
def capture_final_result(self) -> bool: ...
def has_capacity(self) -> bool: ...
def rollout_status(self, rollout_id: str) -> dict[str, Any] | None: ...
def cancel_rollouts(
self,
ids: Sequence[str] | None = None,
prefix: str | None = None,
all: bool = False,
) -> dict[str, str]: ...
def health(self) -> dict[str, Any]: ...
```
| Member | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------ |
| `execute()` | Runs an `AgentWorkflow` and optionally a `Grader` for one request |
| `max_concurrency` | Maximum parallel executions; `0` means no limit |
| `capture_final_result` | Tells the server to retain a backend-computed reward even when no grader callback URL is present |
| `has_capacity()` | Controls whether the server admits another rollout |
| `rollout_status()` | Returns live or recently retained state for one rollout |
| `cancel_rollouts()` | Cancels rollouts selected by IDs, prefix, or all |
| `health()` | Returns backend health information |
`create_rollout_server()` exposes the control methods through HTTP:
| Endpoint | Behavior |
| ---------------------------------- | ----------------------------------------------------------------------------------------------- |
| `POST /rollout` | Returns `202` when admitted; returns `429` with `Retry-After: 5` when `has_capacity()` is false |
| `GET /rollout/{rollout_id}/status` | Returns the backend's current or retained rollout status |
| `POST /rollout/cancel` | Cancels by an exact ID list, an ID prefix, or all in-flight rollouts |
| `GET /health` | Returns `health()` |
Backends can keep the default no-op status and cancellation behavior. `HarborBackend` implements queue admission, status retention, and cancellation.
## LocalBackend
`LocalBackend` executes your workflow and grader directly in the current Python process.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import LocalBackend
backend = LocalBackend(
workflow=MyWorkflow,
workflow_config=my_workflow_config,
grader=MyGrader,
grader_config=my_grader_config,
)
```
Constructor fields:
| Parameter | Type | Description | | |
| ----------------- | ---------------------- | ----------- | ------------------------------------------------------- | -------------------------------------------------------------- |
| `workflow` | \`type\[AgentWorkflow] | str\` | `AgentWorkflow` subclass or `"module:attr"` import path | |
| `workflow_config` | \`AgentWorkflowConfig | str | None\` | Optional config instance or import path passed to the workflow |
| `grader` | \`type\[Grader] | str | None\` | Optional `Grader` subclass or import path |
| `grader_config` | \`GraderConfig | str | None\` | Optional config instance or import path passed to the grader |
Use `LocalBackend` when:
* You want the shortest debug loop.
* You need breakpoints, stack traces, or simple print debugging.
* Your rollout can share the current Python environment.
* You are building a custom harness or eval runner around the SDK.
`LocalBackend` uses `AgentWorkflowConfig.concurrency.max_concurrent` to limit parallel workflow executions. If no workflow config is provided, the default concurrency is `4`.
### Local Error Categories
`LocalBackend` maps workflow and grader exceptions into structured categories:
| Exception | Category |
| ------------------------------------------- | ------------------ |
| `TimeoutError` | `TIMEOUT` |
| `ValueError`, `TypeError`, `AssertionError` | `VALIDATION_ERROR` |
| Other exceptions | `AGENT_ERROR` |
## HarborBackend
`HarborBackend` runs an `AgentWorkflow` or a native Harbor agent inside a Harbor-managed trial environment. It packages workflow projects as wheels, installs them in the task container, and can use either an Osmosis `Grader` or the task's own tests as the reward source.
SDK v0.3 removed the pre-v0.3 `HarborBackend` and renamed `HarborBackendV2` to `HarborBackend`. The new `HarborBackend` has a different constructor and no compatibility alias. Follow the [SDK v0.2 → v0.3 migration guide](/migration-guides/v0-3#harborbackend-users) before upgrading an existing Harbor harness.
Install the Harbor and server features, then import the backend from its Harbor submodule:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install "osmosis-ai[server,harbor]>=0.3,<0.4"
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from pathlib import Path
from harbor.trial.queue import TrialQueue
from osmosis_ai.rollout.backend.harbor import HarborBackend
from osmosis_ai.rollout.server import create_rollout_server
backend = HarborBackend(
orchestrator=TrialQueue(n_concurrent=4),
tasks_dir=Path("tasks/my-task"),
task_mode="template",
agent=MyWorkflow,
grader=MyGrader, # or None to use the task's tests/
max_queue_depth=8,
)
app = create_rollout_server(
backend=backend,
lifespan=backend.prewarm_lifespan(),
)
```
`HarborBackend` is not re-exported from `osmosis_ai.rollout` or `osmosis_ai.rollout.backend`. Use `from osmosis_ai.rollout.backend.harbor import HarborBackend`.
### Constructor
| Parameter | Type / default | Description | | |
| --------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `orchestrator` | `TrialQueue` | Harbor queue used to run trials | | |
| `tasks_dir` | `Path` | A single task directory in template mode, or the root containing task directories in dataset mode | | |
| `agent` | \`type | str\` | `AgentWorkflow` class, `"module:attr"` import path, or registered native agent name | |
| `native_agent_kwargs` | \`dict\[str, Any] | None\` | Extra configuration for a native Harbor agent; invalid for an `AgentWorkflow` | |
| `task_mode` | `"template"` | `"template"` or `"dataset"` | | |
| `model_name` | `"openai/osmosis-rollout"` | Model passed to native agents; request metadata can override it with `harbor_model` | | |
| `grader` | \`type | str | None\` | Optional Osmosis `Grader`; use `None` to score with the task's own `tests/` |
| `workflow_config` | `Any` | Optional workflow config instance or import path bundled with a workflow agent | | |
| `grader_config` | `Any` | Optional grader config instance or import path bundled with the grader | | |
| `code_dir` | \`Path | None\` | Python project to package; defaults to the project containing the workflow or grader | |
| `bundle` | \`Path | None\` | Prebuilt Osmosis bundle wheel to use instead of building from `code_dir` | |
| `environment_config` | \`EnvironmentConfig | None\` | Harbor runtime and placement configuration | |
| `trials_dir` | \`Path | None\` | Host directory for Harbor trial data; defaults to a backend-specific temporary root | |
| `cleanup_successful_trials` | `True` | Removes successful trial staging after artifacts are archived | | |
| `patch_dockerfile_with_sdk` | `None` | Controls whether bundle dependencies are preinstalled into the copied task image; defaults on when a bundle is present | | |
| `agent_setup_timeout_sec` | \`float | None\` | Optional Harbor agent setup timeout | |
| `max_queue_depth` | \`int | None\` | Maximum number of queued rollouts; use an integer >= 1, or `None` for an unbounded queue | |
### Task Modes and Sources
`task_mode` controls how `tasks_dir` is interpreted:
| Mode | `tasks_dir` | Per-request selection |
| ------------ | ----------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `"template"` | One Harbor task directory | The request prompt replaces `instruction.md` in a per-rollout copy |
| `"dataset"` | Root containing one subdirectory per task | Set `metadata["harbor_task_id"]` to a task directory name; the task keeps its own `instruction.md` |
A request can instead set `metadata["harbor_task"]` to fetch a task for that rollout. Supported values are:
* A local path beginning with `.`, `/`, or `~`.
* A Harbor registry package such as `"org/name@ref"`.
* A path inside a Git repository when `metadata["git_url"]` is also set. Pin `metadata["git_commit_id"]` for reproducibility.
A typical task has this layout:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
tasks/my-task/
├── instruction.md
├── environment/
│ └── Dockerfile
└── tests/
└── test.sh
```
In template mode, `instruction.md` can be a placeholder because the rollout prompt replaces it. In dataset mode, each task provides its own instruction.
### Workflow and Native Agents
For an `AgentWorkflow`, the backend builds a wheel from the project containing the class. The project must contain `pyproject.toml` and an importable Python package. Pass `code_dir` when the project cannot be inferred, or pass a prebuilt `bundle` wheel.
The registered native Harbor agent names are:
| Agent | Use |
| ------------------ | ----------------------------------------------------------------------------------------------------------------- |
| `"terminus-2"` | Trainable Harbor terminal agent |
| `"mini-swe-agent"` | Trainable lightweight software-engineering agent |
| `"oracle"` | Runs the task's reference solution to validate tasks and verifiers; does not produce a trainable model trajectory |
Use `native_agent_kwargs` only with a native agent. `model_name` defaults to `openai/osmosis-rollout`, and per-rollout metadata can override it with `metadata["harbor_model"]`.
### Reward Source
Choose one reward path:
| Configuration | Reward source |
| ----------------- | ------------------------------------------------------------------------- |
| `grader=MyGrader` | The bundled Osmosis grader runs as the Harbor verifier |
| `grader=None` | The selected task must provide its own `tests/`, normally `tests/test.sh` |
There is no separate `custom_tests_dir` in v0.3. Put task-native tests under each task's `tests/` directory.
### Prewarming, Capacity, and Cancellation
`prewarm()` builds task images and runs agent setup before serving traffic. Use `prewarm_lifespan()` to connect it to FastAPI startup:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
app = create_rollout_server(
backend=backend,
lifespan=backend.prewarm_lifespan(),
)
```
Template mode prewarms its configured task. Dataset mode requires the task IDs to prewarm:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
app = create_rollout_server(
backend=backend,
lifespan=backend.prewarm_lifespan(["task-a", "task-b"]),
)
```
Set `max_queue_depth` to reject excess work with HTTP `429` instead of queueing without a bound. `rollout_status()` retains recently completed outcomes, and `cancel_rollouts()` can cancel queued or running work by exact IDs, prefix, or all.
### SkyPilot Sandboxes
Harbor trials run in SkyPilot Sandboxes on the Osmosis Platform. Select them through `environment_config`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from harbor.models.environment_type import EnvironmentType
from harbor.models.trial.config import EnvironmentConfig
backend = HarborBackend(
# ...
environment_config=EnvironmentConfig(type=EnvironmentType.SKYPILOT),
)
```
`EnvironmentType.SKYPILOT` is the only value the Osmosis Platform runs. `EnvironmentType.DOCKER` requires a Docker daemon that the managed rollout server does not have, although it remains useful in a self-hosted SDK harness.
Osmosis builds the Dockerfile in the selected task directory and runs the sandbox from it. You do not build or push the image, configure registry credentials, or choose a cluster for a managed run.
**Do not install the `harbor[skypilot]` extra.** It pulls `skypilot-nightly`, which claims the same `sky` namespace as the SkyPilot SDK the rollout server already provides. Install `osmosis-ai[harbor]`; the managed rollout runtime supplies the compatible SkyPilot SDK.
The rollout project declares the SDK and the packages it imports:
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[project]
dependencies = [
"osmosis-ai[server,harbor]>=0.3,<0.4",
]
```
The SkyPilot SDK version is pinned in the platform's rollout image rather than resolved per rollout. Contact Osmosis if you need a newer version.
### Harbor Runtime Context
An `AgentWorkflow` runs inside the Harbor trial environment and receives the standard `AgentWorkflowContext`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
class HarborWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
# This code is already running inside the Harbor trial environment.
...
```
`HarborAgentWorkflowContext` and `ctx.environment` no longer exist. Access files, tools, and processes through normal Python APIs inside `run()`.
## Comparison
| Dimension | LocalBackend | HarborBackend |
| --------------------- | ------------------------------------------- | --------------------------------------------------------- |
| Execution environment | Current Python process | Harbor trial environment |
| Isolation | Shared process and filesystem | Isolated process and filesystem per trial |
| Startup cost | Minimal | Prepares a Harbor task environment and installs the agent |
| Dependencies | Current Python environment | Harbor task environment plus the packaged workflow bundle |
| Debugging | Direct debugger, stack traces, print output | Harbor logs and trial artifacts |
| Queue controls | Backend concurrency limit | Queue depth, status, cancellation, and Harbor concurrency |
| Typical user | SDK harness author, eval tooling | Self-hosted experiments needing isolation |
## Choosing a Backend
Start with `LocalBackend` unless you know you need Harbor isolation. It is faster to debug, has fewer moving parts, and matches the default local starter templates used for smoke tests.
Reach for `HarborBackend` when:
* Untrusted or messy rollout code should not share the host process.
* Tools write files or spawn processes that should be isolated per trial.
* You need a reproducible task environment per rollout execution.
* You want to run native Harbor agents or task-native verifiers.
* You are experimenting with Harbor outside the managed Osmosis training path.
## Relationship to Eval and Training
`osmosis eval submit` and `osmosis train submit` run the rollout server-side and do not expose these SDK backends as CLI options. The rollout entrypoint constructs the backend when its server starts. These SDK-level backends matter when you author an entrypoint or embed `osmosis-ai` in your own harness.
## Next Steps
Update LocalBackend or HarborBackend code from SDK v0.2.
Submit an evaluation run before a training run.
Learn how a workflow registers its sample through a supported integration.
Define the scalar reward for one rollout sample.
Use Strands Agents inside an Osmosis workflow.
Use the OpenAI Agents SDK inside an Osmosis workflow.
# Building Graders
Source: https://docs.osmosis.ai/cli/rollout/graders
Implement the Grader class to define reward signals for training
The `Grader` class defines how your agent's outputs are evaluated and scored. It produces the reward signal that drives reinforcement learning — higher rewards for better outputs, lower rewards for worse ones.
## Grader Base Class
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import Grader, GraderContext
class MyGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
# Evaluate ctx.sample and assign its reward
ctx.set_reward(1.0)
```
The base class signature from the SDK:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class Grader(ABC):
def __init__(self, config: GraderConfig | None = None):
self.config = config
@abstractmethod
async def grade(self, ctx: GraderContext) -> Any:
raise NotImplementedError
```
Like `AgentWorkflow`, the `Grader` has one abstract method — `grade()` — which receives a `GraderContext` containing the agent's outputs and the reference answer for the current dataset row.
## GraderContext
The `ctx` parameter passed to `grade()` provides:
| Field | Type | Description |
| ------------------------ | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ctx.label` | `str \| None` | Reference answer for the current dataset row (typically your `ground_truth` column) |
| `ctx.metadata` | `dict[str, Any] \| None` | Per-row metadata from the dataset's optional `metadata` column. `None` when the row has no metadata. |
| `ctx.sample` | `RolloutSample \| None` | The single agent output, or `None` when the workflow registered no sample source |
| `ctx.project_path` | `str \| None` | Optional project path supplied by the execution harness |
| `ctx.artifacts_dir` | `pathlib.Path \| None` | Per-rollout directory where the grader can write log, trace, and other output files. `None` when the execution environment can't provision a writable directory, so check for `None` before writing files. |
| `ctx.set_reward(reward)` | method | Assign a float reward to `ctx.sample` |
The Grader runs whenever a dataset row has a `label` **or** `metadata`, so you can drive reward signals from metadata alone (for example, expected tool calls or per-row rubrics).
One workflow execution produces at most one sample. Evaluation and training can still execute the workflow multiple times for the same prompt (`[evaluation].n` in evaluation configs, `n_samples_per_prompt` in training configs); each independent execution receives its own `GraderContext`.
### `set_reward`
Call `ctx.set_reward(reward)` to assign a reward to the rollout's sample. The reward should be a float — typically between 0.0 and 1.0, but any float value is accepted.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
ctx.set_reward(0.85)
```
`set_reward` raises a `ValueError` when `ctx.sample` is `None`. Check for a sample before scoring it; a missing sample usually means the workflow did not construct its supported agent or session inside `run()`.
### Writing Artifacts
Use `ctx.artifacts_dir` to persist rubric traces, diffs, or any other files your grader produces. The directory is per-rollout and shared with the workflow that produced the sample, so your grader can also read files the workflow wrote. It's `None` when the environment can't provision one, so guard with `if ctx.artifacts_dir:` before writing to it — an unguarded write raises and fails the grader.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
import json
from osmosis_ai.rollout import Grader, GraderContext
class RubricGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.artifacts_dir:
(ctx.artifacts_dir / "grade_trace.json").write_text(
json.dumps({"reason": "matched rubric"})
)
if ctx.sample is not None:
ctx.set_reward(1.0)
```
After the rollout finishes, collected files appear alongside its sample in the run's **Artifacts** panel on the Osmosis Platform, mirroring the layout you write under `ctx.artifacts_dir`. Artifact collection never affects rewards or rollout status.
### Saved Trajectories
The SDK also saves every finished rollout as an [ATIF](https://www.harborframework.com/docs/agents/trajectory-format) trajectory document next to the artifacts directory (`~/.osmosis//trajectory.json`). Saving runs after your grader completes and is best-effort — it never affects rewards or rollout status.
Your grader sees the framework-native `RolloutSample.messages` on `ctx.sample`. The SDK builds a separate normalized transcript for the ATIF file and does not pass it to grader callbacks. You don't need to do anything to write the file.
## RolloutSample
`ctx.sample` is a `RolloutSample` object containing the AgentWorkflow's output:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from collections.abc import Mapping, Sequence
from typing import Any
from pydantic import BaseModel, Field
class RolloutSample(BaseModel):
messages: Sequence[Mapping[str, Any]] = Field(default_factory=list)
trajectory_messages: Sequence[Mapping[str, Any]] | None = None
label: str | None = None
reward: float | None = None
remove_sample: bool = False
metrics: dict[str, Any] = Field(default_factory=dict)
extra_fields: dict[str, Any] = Field(default_factory=dict)
```
The `messages` list is the conversation your workflow produced for that sample. In many graders, you only need to extract the final answer text from the last assistant message.
For real-world references, see `rollouts/multiply-local-strands/main.py` and `rollouts/multiply-local-openai/main.py` in the `workspace-template` repository. Those files are the source of truth for platform-created workspace repositories.
## Implementation Patterns
### Exact Match Grading
The simplest grading strategy is to compare the agent's final text against `ctx.label`. The helper below extracts text from the last message:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import Grader, GraderContext
def _last_text(sample) -> str:
"""Extract the final text block from a sample's last message."""
if not sample.messages:
return ""
content = sample.messages[-1].get("content", "")
if isinstance(content, str):
return content
if isinstance(content, list):
return next((b["text"] for b in content if isinstance(b, dict) and "text" in b), "")
return ""
class ExactMatchGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
answer = _last_text(ctx.sample).strip()
reward = 1.0 if ctx.label and answer == ctx.label.strip() else 0.0
ctx.set_reward(reward)
```
### LLM-as-Judge Grading
Use a separate LLM to evaluate the quality of agent outputs — useful when correctness is subjective or hard to check programmatically. Unlike the workflow, a grader runs off the training path, so you can call any LLM directly:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
import litellm
from osmosis_ai.rollout import Grader, GraderContext
class LLMJudgeGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
agent_output = _last_text(ctx.sample)
judge_response = await litellm.acompletion(
model="openai/gpt-5.2",
messages=[{
"role": "user",
"content": f"Rate this response from 0.0 to 1.0.\n\n"
f"Expected: {ctx.label}\n"
f"Actual: {agent_output}\n\n"
f"Score (just the number):"
}],
)
score = float(judge_response.choices[0].message.content.strip())
ctx.set_reward(max(0.0, min(1.0, score)))
```
### Tool-Call Based Grading
Evaluate whether the agent made any tool calls, rather than just checking the final text output. Strands records tool invocations as `toolUse` content blocks on assistant messages:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import Grader, GraderContext
class ToolCallGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
used_tool = False
for message in ctx.sample.messages:
if message.get("role") != "assistant":
continue
content = message.get("content") or []
if isinstance(content, list) and any(
isinstance(block, dict) and "toolUse" in block for block in content
):
used_tool = True
break
ctx.set_reward(1.0 if used_tool else 0.0)
```
You can combine multiple grading strategies — for example, check that the agent used the right tools **and** produced a correct final answer, then weight the scores together.
## GraderConfig
Custom grader configs follow the same pattern as `AgentWorkflowConfig` — extend `GraderConfig` and define a module-level config instance in your rollout entrypoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import Grader, GraderConfig, GraderContext
class MyGraderConfig(GraderConfig):
name: str = "my-grader"
partial_credit: bool = True
similarity_threshold: float = 0.8
class MyGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
threshold = self.config.similarity_threshold if self.config else 0.8
# ... use config values in grading logic ...
my_grader_config = MyGraderConfig()
```
Pass the config instance to `LocalBackend(grader_config=my_grader_config)`. Evaluation and training TOML files do not currently set grader config fields directly.
`GraderConfig` extends `BaseConfig` and includes the same `concurrency` field as `AgentWorkflowConfig`, but current backends do not use it to limit grader concurrency. Use evaluation `[evaluation].batch_size`, workflow/backend concurrency, or an explicit limiter inside the grader when your grader calls external services.
| Field | Type | Default | Description |
| ------------- | ------------------- | ---------- | --------------------------------------------------------------------- |
| `name` | `str` | (required) | Identifier for the grader |
| `description` | `str \| None` | `None` | Optional description |
| `concurrency` | `ConcurrencyConfig` | unlimited | Present on the config model; not currently enforced by `LocalBackend` |
## Auto-Discovery
Like `AgentWorkflow`, `osmosis train submit` preflight can discover your `Grader` subclass from the entrypoint module. No registration decorator is needed, but your rollout entrypoint still passes the grader class and optional config to the backend it constructs.
`osmosis train submit` requires a concrete `Grader` in the rollout entrypoint. If the SDK finds no `Grader`, preflight validation fails instead of assigning a default reward.
## Next Steps
Submit an evaluation run to test your AgentWorkflow and Grader before a training run.
# OpenAI Agents Integration
Source: https://docs.osmosis.ai/cli/rollout/openai-agents-integration
Use the OpenAI Agents SDK with Osmosis for training
The Osmosis SDK provides an integration for the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/). Install the `openai-agents` extra (`osmosis-ai[openai-agents]>=0.3,<0.4`) when your rollout uses `Agent`, `Runner.run`, sessions, tools, handoffs, or other OpenAI Agents SDK primitives.
The integration has three main objects:
| Object | Purpose |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `OsmosisAgent` | Drop-in replacement for OpenAI Agents SDK `Agent` that binds an `OsmosisRolloutModel` to the active rollout context |
| `OsmosisRolloutModel` | Placeholder policy model that `OsmosisAgent` replaces with the active rollout model |
| `OsmosisMemorySession` | In-memory session that records the runner conversation and exposes it as a rollout sample for grading |
## Quick Example
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from agents import ModelSettings, Runner
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
from osmosis_ai.rollout.integrations.agents.openai_agents import (
OsmosisAgent,
OsmosisMemorySession,
OsmosisRolloutModel,
)
class OpenAIWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisAgent(
name="assistant",
instructions="Answer the user's request clearly.",
model=OsmosisRolloutModel(),
model_settings=ModelSettings(temperature=1.0, max_tokens=4096),
)
session = OsmosisMemorySession()
await Runner.run(
agent,
ctx.prompt,
session=session,
)
```
Always pass one `OsmosisMemorySession` when using `OsmosisRolloutModel`. The session persists the conversation that your grader will read. Construct it inside `AgentWorkflow.run()` so it registers as the single sample source for the active `RolloutContext`.
## How It Works
`OsmosisAgent` checks whether the `model` argument is an `OsmosisRolloutModel`. If so, it replaces the placeholder with an `OsmosisLitellmModel` bound to the active `RolloutContext`.
`OsmosisMemorySession` registers itself as the single sample source on the current `RolloutContext`.
`Runner.run()` interacts with the session through `get_items()` and `add_items()`. The session stores the persisted OpenAI Agents SDK items in the canonical Responses API shape.
The resolved model sends requests directly to the rollout-scoped Osmosis chat-completions URL.
After `run()` completes, the backend asks the rollout context for its sample. The session returns one `RolloutSample` containing the runner's persisted conversation.
## Complete Example
This example uses an OpenAI Agents SDK tool and a grader that reads the final assistant text from the session-backed sample.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from agents import ModelSettings, Runner, function_tool
from osmosis_ai.rollout import (
AgentWorkflow,
AgentWorkflowContext,
Grader,
GraderContext,
)
from osmosis_ai.rollout.integrations.agents.openai_agents import (
OsmosisAgent,
OsmosisMemorySession,
OsmosisRolloutModel,
)
@function_tool
def multiply(a: int, b: int) -> int:
"""Multiply two integers."""
return a * b
class MultiplyWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisAgent(
name="multiply-agent",
instructions="Use the multiply tool when arithmetic is required.",
model=OsmosisRolloutModel(),
model_settings=ModelSettings(temperature=1.0, max_tokens=4096),
tools=[multiply],
)
session = OsmosisMemorySession()
await Runner.run(
agent,
ctx.prompt,
session=session,
)
def _last_text(sample) -> str:
for item in reversed(sample.messages):
if item.get("role") != "assistant":
continue
content = item.get("content", "")
if isinstance(content, str):
return content
if isinstance(content, list):
for block in content:
if isinstance(block, dict):
text = block.get("text") or block.get("content")
if text:
return text
return ""
class MultiplyGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
answer = _last_text(ctx.sample)
reward = 1.0 if ctx.label and ctx.label.strip() in answer else 0.0
ctx.set_reward(reward)
```
OpenAI Agents SDK sessions persist conversation items rather than Strands-style messages. When writing graders for OpenAI Agents rollouts, inspect `sample.messages` as the runner's persisted session items.
## OsmosisRolloutModel
`OsmosisRolloutModel` is a placeholder. Do not call it directly and do not pass a fixed policy model name into rollout code. In the workspace templates, sampling options live in OpenAI Agents `ModelSettings`.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from agents import ModelSettings
from osmosis_ai.rollout.integrations.agents.openai_agents import (
OsmosisAgent,
OsmosisRolloutModel,
)
agent = OsmosisAgent(
name="assistant",
model=OsmosisRolloutModel(),
model_settings=ModelSettings(
temperature=1.0,
top_p=1.0,
max_tokens=4096,
),
)
```
At runtime, `OsmosisAgent` replaces the placeholder with a model that points at the active Osmosis rollout endpoint.
`OsmosisRolloutModel` is different from the Strands integration's placeholder constructor. For OpenAI Agents examples, use `OsmosisRolloutModel()` with `ModelSettings(...)`, not a `params={...}` dict.
## One Session per Rollout
Use exactly one `OsmosisMemorySession` in each workflow execution that uses `OsmosisRolloutModel`.
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
session = OsmosisMemorySession()
await Runner.run(agent, ctx.prompt, session=session)
```
Create the session inside `run()`. A session created outside the active rollout context cannot be reused inside a rollout run because it was not registered with that context. Constructing a second session in the same execution raises `ValueError`; use handoffs within the same run, or configure multiple independent workflow executions when you need multiple candidate samples.
## Migrating from OpenAI Agents SDK
If you already have an OpenAI Agents SDK workflow, migrate it in four steps:
Change the import and class:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.integrations.agents.openai_agents import OsmosisAgent
```
Then construct `OsmosisAgent(...)` instead of `Agent(...)`.
Replace a fixed model string with an `OsmosisRolloutModel` placeholder:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.integrations.agents.openai_agents import OsmosisRolloutModel
model = OsmosisRolloutModel()
```
Create the session inside `AgentWorkflow.run()` and pass it to `Runner.run()`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
session = OsmosisMemorySession()
await Runner.run(agent, ctx.prompt, session=session)
```
Put the runner call inside an `AgentWorkflow.run()` method. Keep your tools, instructions, handoffs, and agent behavior the same unless they depend on out-of-band state.
## Evaluation
Use the normal eval command:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit configs/eval/.toml
```
During an evaluation run, the platform routes `openai/osmosis-rollout` to the model named in `[experiment].model_path` of the evaluation TOML. During a training run, Osmosis routes the same placeholder to the current training policy.
## Next Steps
Review the shared `AgentWorkflow.run(ctx)` contract.
Write reward logic for the OpenAI Agents session sample.
Submit an evaluation run for your OpenAI Agents rollout before a training run.
# Overview
Source: https://docs.osmosis.ai/cli/rollout/overview
Build custom agent workflows and graders for training on Osmosis
A **rollout** is one execution of agent behavior that Osmosis evaluates during reinforcement learning training. It combines an `AgentWorkflow`, which produces one sample for a prompt, with a `Grader`, which assigns that sample one reward.
Rollouts are normal Python code in your workspace. They can use a simple single-call LLM workflow, a tool-using agent built with Strands Agents, an OpenAI Agents SDK workflow, or a custom harness that you drive through the open source `osmosis-ai` SDK.
## Training Loop
Training on Osmosis repeatedly runs the same four-part loop:
The training cluster selects one row from your dataset and sends its prompt fields to your `AgentWorkflow`. Common datasets contain `system_prompt`, `user_prompt`, and `ground_truth`.
Your workflow receives an `AgentWorkflowContext`, calls the current policy through an Osmosis-supported agent integration, uses any tools you provide, and records one rollout sample.
Your `Grader` receives the sample plus the row's reference answer (`ground_truth`, exposed as `ctx.label`) and assigns one numerical reward.
The reward signal drives the training update, moving the policy toward behavior that receives higher rewards on your task.
This loop is why rollout code must route model calls through Osmosis integrations. The training cluster needs to serve the current policy through the rollout-scoped endpoint, collect traces, and connect the reward to the sample that produced it.
## Files in a Rollout
Each rollout lives under `rollouts/` and is referenced by evaluation and training configs:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
repository/
├── rollouts/
│ └── my-rollout/
│ ├── main.py
│ └── pyproject.toml
├── configs/
│ ├── eval/
│ │ └── my-rollout.toml
│ └── training/
│ └── my-rollout.toml
└── data/
└── test.jsonl
```
| File | Purpose |
| ------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `rollouts/my-rollout/main.py` | Defines exactly one concrete `AgentWorkflow` and one concrete `Grader` for training and evaluation |
| `rollouts/my-rollout/pyproject.toml` | Declares rollout-local Python dependencies |
| `configs/eval/my-rollout.toml` | Points the evaluation run at the rollout, entrypoint, evaluation policy model, and platform dataset |
| `configs/training/my-rollout.toml` | Points the training run at the rollout code version and training settings |
`osmosis train submit` and `osmosis eval submit` both discover rollout classes from the entrypoint file and run the rollout server-side. Keep helper classes, tools, and config objects wherever you like, but expose the concrete `AgentWorkflow` and `Grader` classes that the platform should validate.
## Core Abstractions
| Abstraction | What it does | Where to learn more |
| ----------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `AgentWorkflow` | Defines agent behavior: prompt handling, model calls, tool use, and sample creation | [Building AgentWorkflows](/cli/rollout/agent-workflows) |
| `Grader` | Defines reward logic: exact matching, programmatic checks, LLM-as-judge, or custom scoring | [Building Graders](/cli/rollout/graders) |
| Agent integration | Connects your agent framework to the active Osmosis rollout context | [Strands Integration](/cli/rollout/strands-integration), [OpenAI Agents Integration](/cli/rollout/openai-agents-integration) |
| Execution backend | Runs rollout code in-process or in a Harbor-managed environment when you drive the SDK yourself | [Execution Backends](/cli/rollout/execution-backends) |
## Choose an Agent Framework
Most rollout authors start with one of the built-in agent integrations:
Use `OsmosisStrandsAgent` when you want Strands tools, Strands message handling, and a direct migration path from an existing Strands `Agent`.
Use `OsmosisAgent` when your workflow already uses the OpenAI Agents SDK, `Runner.run`, sessions, handoffs, or OpenAI-style tool orchestration.
Both integrations use an `OsmosisRolloutModel` placeholder. You do not hard-code the training model inside rollout code; Osmosis resolves the placeholder to the current policy at runtime.
Do not call provider SDKs directly from `AgentWorkflow.run()` with a fixed model such as `openai/gpt-5.2`. Direct calls bypass the active `RolloutContext`, so the platform cannot route policy requests, collect the sample, or connect its reward to the right rollout.
## Choose an Execution Backend
If you use `osmosis eval submit` or `osmosis train submit`, the platform manages execution and you do not choose a backend from the CLI. The entrypoint decides which SDK backend to construct when the rollout server starts, and the starter templates use `LocalBackend` unless you choose the Harbor template.
If you build on the Harbor template, run trials in SkyPilot Sandboxes. The Osmosis Platform does not support Docker-backed Harbor execution.
You only choose a backend explicitly when embedding the open source SDK in your own harness:
| Backend | Use when |
| --------------- | -------------------------------------------------------------------------------------- |
| `LocalBackend` | You want fast in-process execution, easy debugging, and no Docker dependency |
| `HarborBackend` | You need Harbor-managed per-trial isolation, with trials running in SkyPilot Sandboxes |
See [Execution Backends](/cli/rollout/execution-backends) for SDK-level examples and tradeoffs.
Upgrading an SDK harness from v0.2? `LocalBackend` keeps its constructor, while v0.3 replaces the pre-v0.3 `HarborBackend` with the implementation previously named `HarborBackendV2`. Follow the [SDK v0.2 → v0.3 migration guide](/migration-guides/v0-3) before changing dependencies.
## Start from a Template
If you already have a task or dataset, start with [Create Your Own Rollout](/platform/create-your-own-rollout). Platform-created workspace repositories include project-local Agent Skills that guide an AI coding agent through dataset planning, rollout creation, evaluation runs, debugging, and training run readiness.
List available starter templates:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template list
```
Apply a Strands starter:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template apply multiply-local-strands
```
Or apply an OpenAI Agents starter:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template apply multiply-local-openai
```
Templates are copied from the platform workspace template repository. They write rollout code under `rollouts/` plus matching evaluation and training configs, and are the quickest way to see the expected file layout, dependency declaration, and end-to-end workflow.
## Path to Training
Once your rollout exists, use this path:
Put policy calls behind `OsmosisStrandsAgent` or `OsmosisAgent`, pass the dataset prompt from `ctx.prompt`, and keep any task-specific tools close to the rollout.
Score `ctx.sample` and call `ctx.set_reward()` once. Start with a deterministic grader when possible, then add LLM-as-judge logic only when the task is subjective.
Push rollout changes to the branch you intend to run. [Git Sync](/cli/workspace/git-sync) catalogs every branch; set `branch` in the evaluation or training config to select a feature branch, or omit it to use the default branch.
Run `osmosis eval submit configs/eval/my-rollout.toml` and inspect rewards, failures, and per-row results with `osmosis eval info `. See [Evaluation](/cli/rollout/eval).
Run `osmosis train submit configs/training/my-rollout.toml`. See [Training Runs](/platform/training-runs) for submission behavior.
## Next Steps
Use project-local Agent Skills to create a task-specific rollout with evaluation run gates.
Learn the `AgentWorkflow.run(ctx)` contract and common implementation patterns.
Define reward signals that can drive training.
Build tool-using rollouts with AWS Strands Agents.
Build rollouts with the OpenAI Agents SDK.
# Strands Integration
Source: https://docs.osmosis.ai/cli/rollout/strands-integration
Use the Strands agent framework with Osmosis for training
[Strands Agents](https://github.com/strands-agents/sdk-python) is an AWS agent framework for building tool-using agents. Use the Osmosis Strands integration when you want Strands tools, Strands message handling, and a direct migration path from an existing `strands.Agent`.
Install the `strands` extra when a rollout uses this integration: `osmosis-ai[strands]>=0.3,<0.4`.
## Integration Objects
| Object | Purpose |
| --------------------- | ------------------------------------------------------------------------------------------------- |
| `OsmosisStrandsAgent` | Drop-in replacement for Strands `Agent` that registers the sample with the active rollout context |
| `OsmosisRolloutModel` | Placeholder model that resolves to the current Osmosis policy at runtime |
`OsmosisStrandsAgent` preserves normal Strands constructor arguments such as `tools`, `system_prompt`, `messages`, and callback handlers.
## Quick Example
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
from osmosis_ai.rollout.integrations.agents.strands import (
OsmosisRolloutModel,
OsmosisStrandsAgent,
)
class StrandsWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisStrandsAgent(
name="assistant",
model=OsmosisRolloutModel(params={"temperature": 1.0}),
messages=ctx.prompt,
callback_handler=None,
)
await agent.invoke_async()
```
`ctx.prompt` is already the ready-to-use input for the current sample. If your dataset row contains `system_prompt` and `user_prompt`, the SDK assembles those fields before your workflow runs.
## Tools
Define Strands tools normally with `@tool`, then pass them to `OsmosisStrandsAgent`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from strands import tool
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
from osmosis_ai.rollout.integrations.agents.strands import (
OsmosisRolloutModel,
OsmosisStrandsAgent,
)
@tool
def search(query: str) -> str:
"""Search for information."""
return f"Results for: {query}"
class SearchWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisStrandsAgent(
name="search-agent",
model=OsmosisRolloutModel(params={"temperature": 1.0}),
tools=[search],
system_prompt="You are a helpful assistant.",
messages=ctx.prompt,
callback_handler=None,
)
await agent.invoke_async()
```
For most tool-using agents, one `invoke_async()` call is enough because Strands handles the model-tool loop internally. Add an outer loop only when you need extra stopping conditions or a hard cap across repeated invocations.
## OsmosisRolloutModel
`OsmosisRolloutModel` does not take a `model_id`. The SDK uses the placeholder model id `openai/osmosis-rollout` at runtime, and Osmosis routes it to the current policy.
Use the `params` dict for sampling options:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
model = OsmosisRolloutModel(
params={
"temperature": 1.0,
"max_tokens": 1024,
}
)
```
Construct `OsmosisStrandsAgent` inside `AgentWorkflow.run()` or another path where the execution backend has already installed an active `RolloutContext`. Constructing it at module import time will fail because no rollout context exists yet.
## How Sample Collection Works
When constructed with an `OsmosisRolloutModel`, `OsmosisStrandsAgent` performs these steps:
It reads the active `RolloutContext` from the current execution scope and raises `RuntimeError` if none is available.
It creates a LiteLLM model connected directly to the rollout-scoped Osmosis chat-completions URL.
It registers itself with the rollout context so the backend can collect the Strands message history as a `RolloutSample`.
It delegates to the normal Strands `Agent` constructor with the resolved model. Tools, prompts, messages, and callbacks pass through unchanged.
## Complete Example
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from strands import tool
from osmosis_ai.rollout import (
AgentWorkflow,
AgentWorkflowContext,
Grader,
GraderContext,
)
from osmosis_ai.rollout.integrations.agents.strands import (
OsmosisRolloutModel,
OsmosisStrandsAgent,
)
@tool
def calculator(expression: str) -> str:
"""Evaluate a math expression."""
return str(eval(expression))
class MathWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisStrandsAgent(
name="math-agent",
model=OsmosisRolloutModel(params={"temperature": 1.0}),
tools=[calculator],
system_prompt="You are a math assistant. Use the calculator tool.",
messages=ctx.prompt,
callback_handler=None,
)
await agent.invoke_async()
def _last_text(sample) -> str:
if not sample.messages:
return ""
content = sample.messages[-1].get("content", "")
if isinstance(content, str):
return content
if isinstance(content, list):
return next((b["text"] for b in content if isinstance(b, dict) and "text" in b), "")
return ""
class MathGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
answer = _last_text(ctx.sample)
reward = 1.0 if ctx.label and ctx.label.strip() in answer else 0.0
ctx.set_reward(reward)
```
## Migrating from Strands Agent
If you already have a Strands agent, migrate it in four steps:
Replace `from strands import Agent` with:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.integrations.agents.strands import OsmosisStrandsAgent
```
Replace your fixed LiteLLM model with an Osmosis placeholder:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.integrations.agents.strands import OsmosisRolloutModel
model = OsmosisRolloutModel(params={"temperature": 1.0})
```
Drop the `model_id`; the training cluster decides which policy to serve.
Replace `Agent(...)` with `OsmosisStrandsAgent(...)`. Keep tools, system prompt, messages, and callbacks the same.
Move the agent construction and `await agent.invoke_async()` call into `AgentWorkflow.run()`.
## Strands vs OpenAI Agents
| Choose Strands when | Choose OpenAI Agents when |
| ---------------------------------------------------- | ------------------------------------------------------------------------- |
| Your tools are already Strands `@tool` functions | Your workflow already uses `Runner.run` and OpenAI Agents SDK sessions |
| You want Strands message traces in `sample.messages` | You want persisted Responses API-style session items in `sample.messages` |
| You are migrating from `strands.Agent` | You are migrating from OpenAI Agents SDK `Agent` |
See [OpenAI Agents Integration](/cli/rollout/openai-agents-integration) for the OpenAI Agents SDK path.
## Next Steps
Review the shared workflow contract.
Score the sample produced by your Strands agent.
Submit an evaluation run for your Strands rollout.
Compare the OpenAI Agents SDK integration.
# Git Sync
Source: https://docs.osmosis.ai/cli/workspace/git-sync
Sync rollout code and configs from your workspace repository to Osmosis
## Overview
Git Sync connects a platform workspace to a GitHub workspace repository. The platform reads rollout code and configuration from that repository, discovers rollouts under `rollouts/`, and makes them available for training.
Your local workspace directory is a clone of the same repository. Push to any branch to sync its rollout catalog.
## Repository Setup
Workspace owners and admins connect GitHub in the platform, then create a private workspace repository from the Osmosis workspace template or connect an existing repository.
New repositories created from the platform include the starter examples `multiply-local-strands`, `multiply-local-openai`, and `multiply-harbor-strands`. For the full first-time setup flow, start with [Onboarding](/platform/onboarding).
Your local workspace directory should be a clone of the connected GitHub repository. Invited members use the existing workspace repository rather than creating a second one.
From your local workspace directory, confirm the `origin` remote points at the connected GitHub repository:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git remote get-url origin
osmosis doctor
```
## How Sync Works
* **Pushing to any branch** triggers an automatic sync for that branch.
* The platform discovers rollout definitions in the `rollouts/` directory independently for each branch.
* Each rollout subdirectory (e.g. `rollouts/my-rollout/`) becomes a rollout entity on the platform, available for training and evaluation.
* Config files under `configs/` are also synced so that `osmosis train submit` can reference them by path.
* The platform records sync history with branch, commit SHA, author, message, status, and number of rollouts discovered.
* Deleting a branch removes that branch's rollout catalog without removing rollouts that still exist on another branch.
Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set `branch` to use a pushed branch or `commit_sha` for a specific pushed revision; omit both to use the default branch.
## Sync Status
View sync history in the platform under **Git Integration**. Each sync event shows:
| State | Meaning |
| ----------- | -------------------------------------------------- |
| **Pending** | Push detected, sync queued |
| **Syncing** | Platform is processing the repository contents |
| **Success** | All rollouts synced successfully |
| **Failed** | Sync encountered an error (check logs for details) |
A manual sync action is available in the platform to re-process every repository branch without pushing a new commit.
## Blocking States
The platform shows a banner when GitHub setup blocks training workflows.
| State | Effect | Fix |
| ----------------------- | --------------------------------------------------- | --------------------------------------------------- |
| No GitHub installation | Rollout sync and training setup cannot proceed. | Connect GitHub from **Git Integration**. |
| No repository connected | The platform has no workspace repository to sync. | Create or connect a workspace repository. |
| GitHub App disconnected | Training runs are blocked until access is restored. | Reconnect the GitHub App. |
| Repository needs setup | The repository record needs repair or replacement. | Open **Git Integration** and follow the setup flow. |
## Important Notes
* Every branch is synced automatically; there is no per-workspace branch-sync setting.
* The **Rollouts** page has a branch picker. The repository default branch is selected first, and `?branch=` preserves the selection in the URL.
* If the repository is renamed on GitHub, update your local `origin` remote and check the platform connection.
Use `branch` in a [run config](/cli/config-files) to follow a branch, or `commit_sha` to pin a specific commit. They are mutually exclusive; omit both to use the default branch.
## Next Steps
Learn how local CLI commands are scoped to a repository.
Submit training after Git Sync succeeds.
# Overview
Source: https://docs.osmosis.ai/cli/workspace/overview
Understand workspace repositories and local workspace directories
## What Is a Workspace?
An Osmosis workspace is the platform space where your team manages datasets, training runs, models, deployments, members, and one connected GitHub repository.
The connected GitHub repository is the **workspace repository**. Your local clone is the **workspace directory**.
| Term | Meaning |
| ------------------------ | ----------------------------------------------------------- |
| **Platform workspace** | The team space in the Osmosis Platform. |
| **Workspace repository** | The GitHub repository connected to that platform workspace. |
| **Workspace directory** | Your local clone of the workspace repository. |
## Getting Started
Start with [Onboarding](/platform/onboarding) to create or join a platform workspace, connect GitHub, clone the workspace repository, and authenticate the CLI.
After onboarding, your local workspace directory contains the standard workspace structure:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
repository/
├── rollouts/ # AgentWorkflow + Grader code
├── configs/ # Training and evaluation configs
├── data/ # Local test datasets
├── pyproject.toml # Python project config
├── AGENTS.md # AI coding assistant instructions
├── CLAUDE.md # Claude Code instructions
└── .gitignore
```
See [Structure & Configuration](/cli/workspace/structure-and-config) for a detailed breakdown.
## CLI Context
The CLI scopes platform commands to the workspace repository you are currently inside. It reads Git `origin`, resolves the GitHub `owner/repo`, and sends that repository identity to the platform.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git remote get-url origin
osmosis dataset list
```
If the CLI cannot resolve the workspace, run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
```
## Typical Local Loop
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis rollout init
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git add .
git commit -m "add rollout"
git push
```
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit configs/eval/.toml
```
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train submit configs/training/.toml
```
## Next Steps
Learn how GitHub `origin` scopes CLI commands to a platform workspace.
Understand the files in a cloned workspace repository.
Push to GitHub and let the platform sync rollout code automatically.
# Workspace Repository
Source: https://docs.osmosis.ai/cli/workspace/repository
Understand how a GitHub repository connects your local CLI commands to an Osmosis workspace
A workspace repository is the GitHub repository connected to one Osmosis platform workspace. The platform creates it from the Osmosis workspace template or connects an existing repository, then uses Git Sync to discover rollouts and configs.
Your local workspace directory is a clone of that repository.
## How CLI Context Works
When you run platform commands from inside a workspace directory, the CLI:
1. Finds the Git worktree root.
2. Reads the `origin` remote.
3. Normalizes the GitHub repository identity as `owner/repo`.
4. Sends that identity to the platform so the request is scoped to the matching workspace.
This means commands like `osmosis dataset list`, `osmosis train submit`, `osmosis rollout list`, and `osmosis model deploy` should be run from inside the cloned workspace repository.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git remote get-url origin
osmosis dataset list
```
The `origin` remote must point to a GitHub repository connected to your platform workspace. If the repository was renamed on GitHub, update your local remote before running CLI commands.
## Common Requirements
| Requirement | Why it matters |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Git worktree | The CLI uses the Git root as the workspace directory. |
| GitHub `origin` remote | The CLI maps `origin` to the platform workspace. |
| Platform login | Platform commands require `osmosis auth login` or `OSMOSIS_TOKEN`. |
| Required directories | `rollouts/`, `configs/training/`, `configs/eval/`, and `data/` make the directory a valid Osmosis workspace directory. |
Run a local health check when something looks wrong:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
```
Repair missing scaffold directories:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor --fix
```
## Repository Ownership
Workspace creators usually create the repository from the platform during onboarding. Invited members should clone the existing workspace repository rather than creating a separate one.
Only workspace owners and admins can manage GitHub repository connection settings in the platform. Members can clone and use the repository if they have GitHub access.
## Related Commands
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template list
osmosis template apply multiply-local-strands
osmosis rollout init my-rollout
osmosis eval submit configs/eval/my-rollout.toml
osmosis train submit configs/training/my-rollout.toml
```
## Next Steps
Learn the repository layout and config directories.
Understand how pushes become platform rollouts.
# Structure & Configuration
Source: https://docs.osmosis.ai/cli/workspace/structure-and-config
Understand the workspace repository layout and configuration files
A workspace directory is a local clone of the GitHub workspace repository connected to your platform workspace. The platform creates new workspace repositories from the Osmosis workspace template.
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
repository/
├── rollouts/ # AgentWorkflow + Grader code
├── configs/
│ ├── eval/ # Evaluation configs
│ ├── AGENTS.md # AI assistant instructions for configs
│ └── training/
│ └── default.toml # Training config template
├── data/ # Local test datasets
├── pyproject.toml # Python project config
├── README.md # Project readme
├── AGENTS.md # AI coding assistant instructions
├── CLAUDE.md # Claude Code instructions
└── .gitignore # Git ignore rules
```
***
## Required Directories
The CLI expects these directories to exist:
| Path | Purpose |
| ------------------- | --------------------------------------------------- |
| `rollouts/` | Rollout code, one subdirectory per rollout |
| `configs/training/` | Training run TOML files for `osmosis train submit` |
| `configs/eval/` | Evaluation run TOML files for `osmosis eval submit` |
| `data/` | Local datasets used by `osmosis dataset upload` |
Run a health check from anywhere inside the workspace directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
```
Repair missing scaffold directories without overwriting existing files:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor --fix
```
## `rollouts/`
The directory where your AgentWorkflow and Grader code lives. Each rollout is a subdirectory containing an entrypoint file (typically `main.py`) that defines the agent workflow and grading logic.
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
rollouts/
└── my-rollout/
├── main.py # Entrypoint: defines AgentWorkflow + Grader
├── pyproject.toml # Rollout package dependencies
└── README.md # Rollout notes
```
Create a new rollout scaffold:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis rollout init
```
Or apply a starter template:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis template list
osmosis template apply multiply-local-strands
```
## `configs/`
Configuration files for the two CLI operations: training and evaluation.
| Subdirectory | Purpose | Reference |
| ------------------- | ------------------------------------------------ | ---------------------------------------------------- |
| `configs/training/` | Training run configs for `osmosis train submit` | [Training Config](/cli/config-files#training-config) |
| `configs/eval/` | Evaluation run configs for `osmosis eval submit` | [Evaluation Config](/cli/config-files#eval-config) |
The `configs/training/default.toml` template and rollout-specific evaluation configs named `configs/eval/.toml` are pre-populated with required fields and commented-out optional settings. See [Configuration Files](/cli/config-files) for the full TOML schema reference.
## `data/`
Directory for local dataset files. Upload them to the platform with `osmosis dataset upload`, then reference the uploaded dataset by name from your [training](/cli/config-files#training-config) and [evaluation](/cli/config-files#eval-config) configs.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset upload data/.jsonl
```
## `pyproject.toml`
Standard Python project configuration for the workspace repository. Individual rollouts can also have their own `rollouts//pyproject.toml` files for rollout-specific dependencies.
```toml pyproject.toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[project]
name = "osmosis-workspace"
description = "Osmosis workspace repository"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"osmosis-ai[server,strands]>=0.3,<0.4",
]
```
## `AGENTS.md` & `CLAUDE.md`
Instruction files for AI coding assistants. `AGENTS.md` provides general guidance for any AI assistant (GitHub Copilot, Cursor, etc.), while `CLAUDE.md` contains Claude Code-specific instructions.
Both files describe the workspace structure, conventions, and Osmosis-specific patterns so your AI assistant can effectively help you write rollout code.
## Generated Runtime State
The CLI may create local runtime files under `.osmosis/`, such as exported metrics. Treat those as local state, not source code.
## Next Steps
Learn how the CLI resolves the connected workspace.
Review training and evaluation TOML schemas.
# Introduction
Source: https://docs.osmosis.ai/introduction
Osmosis is a post-training platform for LLMs. The Osmosis CLI abstracts away the infrastructure challenges of distributed training & RL pipeline design.
You can define and/or port in your agent loop, tools, rewards, and training data. Osmosis handles the rest to deliver task-specific models that can outperform foundation models on performance, cost, and latency.
## Get Started
Set up your platform workspace, GitHub repository, local clone, and CLI session.
After onboarding, run a known-good example from evaluation run to training run.
Use your AI coding agent to turn a task or dataset into a validated rollout.
Understand workspaces, training runs, metrics, deployments, and model management.
## Use Cases
Build domain-specific extraction models to capture the exact structure and content for any document at a fraction of the cost of a foundation model or managed product.
Teach AI agents to use the exact tools they'll have in production. Osmosis powers AI agents that stay reliable, even in the most complex multi-step, multi-tool tasks.
Train specialized coding models for blazing fast generation of domain-specific languages, front-end components, and tests — without needing a large model.
## Why Osmosis
Osmosis ships primitives and tool modules into the platform so coding agents like Claude Code, Codex, and others can start, monitor, and iterate on training runs.
Osmosis implements and handles the RL algorithms and infrastructure that enable performant, GPU-efficient training runs.
Osmosis integrates with your evaluation solutions and coding agents to automatically start re-training runs without the need for an engineer in the loop.
# SDK v0.2 → v0.3 Migration Guide
Source: https://docs.osmosis.ai/migration-guides/v0-3
Migrate LocalBackend and HarborBackend rollouts from Osmosis SDK v0.2 to v0.3
Osmosis SDK v0.3 contains two independent breaking changes. All rollouts move to a one-execution, one-sample, one-reward contract. Separately, the pre-v0.3 `HarborBackend` is removed and `HarborBackendV2` becomes the new `HarborBackend` with a different constructor and execution model.
Upgrade the package requirement, workflow, grader, integrations, and backend entrypoint together. Mixing v0.2 rollout code with a v0.3 backend usually fails at runtime.
Use the section that matches your backend:
| Current backend | Migration path |
| ------------------------ | --------------------------------------------------------------------------------------------------------- |
| `LocalBackend` | Keep its constructor, then update the shared sample, reward, integration, and routing APIs |
| Pre-v0.3 `HarborBackend` | Apply the shared API changes, then replace the backend constructor and container execution model |
| `HarborBackendV2` | Apply the shared API changes and rename it to `HarborBackend`; its v2 constructor is the v0.3 constructor |
Before changing code:
* Keep the last successful v0.2 evaluation run for comparison.
* Use Python 3.12 or later.
* Recreate the rollout environment and lockfile after changing the SDK requirement.
* Search for removed APIs before and after the migration.
## LocalBackend Users
`LocalBackend` keeps the same keyword-only constructor in v0.3:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout import LocalBackend
backend = LocalBackend(
workflow=MyWorkflow,
workflow_config=my_workflow_config,
grader=MyGrader,
grader_config=my_grader_config,
)
```
The migration work is in the workflow, grader, integrations, and any custom protocol code around the backend.
### 1. Upgrade the Package and Select Features
The base distribution contains the CLI and framework-neutral rollout core. Add only the extras your rollout imports:
```bash Strands theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade "osmosis-ai[strands]>=0.3,<0.4"
```
```bash OpenAI Agents theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade "osmosis-ai[openai-agents]>=0.3,<0.4"
```
```bash Rollout server theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade "osmosis-ai[server]>=0.3,<0.4"
```
```bash Everything theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade "osmosis-ai[full]>=0.3,<0.4"
```
Combine extras when needed. For example, a server entrypoint using Strands should declare:
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[project]
dependencies = [
"osmosis-ai[server,strands]>=0.3,<0.4",
]
```
Other extras include `openai-agents`, `harbor`, `rubric`, and `parquet`.
### 2. Move from Many Samples to One Sample
Each workflow execution now produces at most one `RolloutSample`, and its grader assigns one scalar reward.
| v0.2 | v0.3 | Action |
| ------------------------------------------ | ------------------------------- | ---------------------------------------------------------- |
| `GraderContext.samples` | `GraderContext.sample` | Read and score the single sample |
| `ctx.set_sample_reward(sample_id, reward)` | `ctx.set_reward(reward)` | Remove the sample ID argument |
| `register_sample_source(name, source)` | `set_sample_source(source)` | Register exactly one source |
| `get_samples()` | `get_sample()` | Return one `RolloutSample` or `None` |
| `RolloutSample.id` | Removed | Use the rollout identity supplied by the execution URLs |
| `MultiTurnMode` | Removed | Keep the conversation history inside the single sample |
| Multiple registered agents or sessions | One registered agent or session | Run independent candidates as separate workflow executions |
Update graders from a loop over samples to one explicit sample check:
```python v0.2 theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class ExactMatchGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
for sample_id, sample in ctx.samples.items():
answer = last_text(sample)
reward = 1.0 if answer == ctx.label else 0.0
ctx.set_sample_reward(sample_id, reward)
```
```python v0.3 theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class ExactMatchGrader(Grader):
async def grade(self, ctx: GraderContext) -> None:
if ctx.sample is None:
raise ValueError("workflow produced no sample")
answer = last_text(ctx.sample)
reward = 1.0 if answer == ctx.label else 0.0
ctx.set_reward(reward)
```
`set_reward()` raises `ValueError` when the workflow produced no sample. Raising an explicit error before scoring generally gives a clearer evaluation failure.
### 3. Register One Agent or Session per Execution
Construct exactly one registered `OsmosisStrandsAgent` or `OsmosisMemorySession` inside `AgentWorkflow.run()`. A second registration raises `ValueError`.
For OpenAI Agents, the v0.3 session has no name or sample-ID argument:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class OpenAIWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
agent = OsmosisAgent(
name="assistant",
instructions="Answer the user's request clearly.",
model=OsmosisRolloutModel(),
)
session = OsmosisMemorySession()
await Runner.run(agent, ctx.prompt, session=session)
```
Handoffs and tool calls can remain inside that one agent run. If you need several candidate answers for one prompt, configure evaluation or training to execute the workflow several times.
### 4. Update Custom Sample Sources
Built-in integrations already use the v0.3 API. A custom integration must implement the singular source contract:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
class MySampleSource(SampleSource):
async def get_sample(self) -> RolloutSample:
return RolloutSample(messages=self.messages)
rollout_ctx.set_sample_source(MySampleSource(messages))
sample = await rollout_ctx.get_sample()
```
Do not set an `id` on `RolloutSample`. To control the normalized ATIF transcript, set `trajectory_messages`; setting it to `None` disables trajectory persistence for that sample.
### 5. Update Custom Routing and Backend Adapters
Skip this step if you only use the built-in integrations and `LocalBackend`.
Treat the chat-completions and callback URLs supplied for an execution as opaque, rollout-scoped endpoints. Do not append a rollout ID or attach the removed `x-sample-id` and `x-rollout-id` routing headers.
Custom backends return one `sample` in `ExecutionResult`. If you read the container exchange files directly, update readers to the singular file names and payloads:
```json sample.json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{
"messages": [],
"reward": null,
"remove_sample": false,
"metrics": {},
"extra_fields": {}
}
```
```json reward.json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{
"reward": 1.0
}
```
### 6. Verify a LocalBackend Migration
1. Confirm that removed APIs no longer appear in the rollout:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
rg "ctx\.samples|set_sample_reward|register_sample_source|get_samples|MultiTurnMode|x-sample-id|x-rollout-id" rollouts
```
2. Recreate the rollout environment so its lockfile contains v0.3 and the selected extras.
3. Run the workflow locally and confirm that each successful execution produces one sample.
4. Submit an evaluation run and confirm that each graded sample has one scalar reward.
5. Compare rewards and final messages with the last successful v0.2 evaluation before submitting training.
### Common LocalBackend Problems
The workflow execution already registered its sample source. Reuse one agent or session for the conversation, or move independent candidates into separate workflow executions.
Construct the supported agent or session inside `AgentWorkflow.run()`. Objects created at module import time cannot register with the active rollout context.
Install the matching extra in the rollout-local environment and regenerate its lockfile.
Pass the supplied chat-completions URL directly to the integration. Remove code that reconstructs the URL, adds a rollout path segment, or attaches legacy routing headers.
## HarborBackend Users
Harbor users must first apply the shared workflow, grader, sample-source, and routing changes in [LocalBackend Users](#localbackend-users). Then migrate the Harbor class and constructor.
The name `HarborBackend` still imports in v0.3, but it now refers to the implementation previously called `HarborBackendV2`. A pre-v0.3 call using `task_dir`, `user_code_dir`, or `workflow` raises `TypeError`; there is no legacy compatibility mode.
### 1. Upgrade the Package and Import
Install the v0.3 Harbor and server features:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install --upgrade "osmosis-ai[server,harbor]>=0.3,<0.4"
```
Install `osmosis-ai[harbor]`, not Harbor's `skypilot` extra. The managed rollout runtime supplies the compatible SkyPilot SDK.
Use the Harbor submodule import:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.backend.harbor import HarborBackend
```
If you already use `HarborBackendV2`, change only the class name and keep its v2 constructor arguments:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# v0.2 preview API
from osmosis_ai.rollout.backend.harbor import HarborBackendV2
# v0.3
from osmosis_ai.rollout.backend.harbor import HarborBackend
```
`HarborBackendV2` is not retained as an alias.
### 2. Replace the Pre-v0.3 Constructor
The old backend mounted your source tree and SDK into a task environment. The v0.3 backend packages the workflow project into a wheel and installs it inside the task container.
```python Pre-v0.3 theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from pathlib import Path
from osmosis_ai.rollout.backend.harbor import HarborBackend
backend = HarborBackend(
orchestrator=trial_queue,
task_dir=Path("tasks/my-task"),
user_code_dir=Path("."),
workflow=MyWorkflow,
workflow_config=my_workflow_config,
grader=MyGrader,
grader_config=my_grader_config,
cleanup_successful_trials=True,
)
```
```python v0.3 theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from pathlib import Path
from osmosis_ai.rollout.backend.harbor import HarborBackend
backend = HarborBackend(
orchestrator=trial_queue,
tasks_dir=Path("tasks/my-task"),
task_mode="template",
agent=MyWorkflow,
workflow_config=my_workflow_config,
grader=MyGrader,
grader_config=my_grader_config,
code_dir=Path("."),
cleanup_successful_trials=True,
)
```
Migrate every old constructor parameter with this table:
| Pre-v0.3 parameter | v0.3 replacement | Migration notes |
| ---------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------- |
| `orchestrator=` | `orchestrator=` | Carries over unchanged |
| `task_dir=` | `tasks_dir=` and `task_mode=` | Use `"template"` for one reusable task or `"dataset"` for a directory of tasks |
| `user_code_dir=` | `code_dir=` or `bundle=` | Point `code_dir` at a package project with `pyproject.toml`, or provide a prebuilt bundle wheel |
| `workflow=` | `agent=` | Pass an `AgentWorkflow` class/import path or a registered native Harbor agent name |
| `workflow_config=` | `workflow_config=` | Carries over; it is packaged with a workflow agent |
| `grader=` | `grader=` | Carries over; the grader is packaged and runs as the Harbor verifier |
| `grader_config=` | `grader_config=` | Carries over unchanged |
| `trials_dir=` | `trials_dir=` | Carries over, but omitting it now uses a backend-specific temporary root instead of `Path("trials")` |
| `custom_tests_dir=` | Removed | Put tests under each task's `tests/`, or pass an Osmosis `grader=` |
| `environment_config=` | `environment_config=` | Carries over unchanged |
| `prebuild_local_image=` | Removed | Harbor handles image caching; call `prewarm()` or `prewarm_lifespan()` before serving |
| `symlink_environment=` | Removed | Each rollout materializes a task copy and relies on Harbor image caching |
| `cleanup_successful_trials=` | `cleanup_successful_trials=` | Carries over unchanged |
Remove the private `_sdk_source_dir` argument if your harness used it.
The v0.3 constructor also adds these controls:
| New parameter | Use |
| --------------------------- | ------------------------------------------------------------------------- |
| `native_agent_kwargs` | Configure a registered native Harbor agent |
| `model_name` | Select the native agent model; defaults to `openai/osmosis-rollout` |
| `bundle` | Reuse a prebuilt Osmosis bundle wheel instead of building from `code_dir` |
| `patch_dockerfile_with_sdk` | Control preinstallation of bundle dependencies in the task image |
| `agent_setup_timeout_sec` | Limit Harbor agent setup time |
| `max_queue_depth` | Bound queued rollouts and enable `429` admission control |
### 3. Choose a Task Mode
For the closest replacement of the old single `task_dir`, use template mode:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
backend = HarborBackend(
orchestrator=trial_queue,
tasks_dir=Path("tasks/my-task"),
task_mode="template",
agent=MyWorkflow,
)
```
The request prompt replaces `instruction.md` in a per-rollout copy of that task.
Use dataset mode when `tasks_dir` contains one directory per task:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
backend = HarborBackend(
orchestrator=trial_queue,
tasks_dir=Path("tasks"),
task_mode="dataset",
agent=MyWorkflow,
)
```
Each request must set `metadata["harbor_task_id"]`; the selected task keeps its own `instruction.md`.
For either mode, `metadata["harbor_task"]` can select a per-rollout local path, Harbor registry package such as `"org/name@ref"`, or Git task. Git tasks also set `metadata["git_url"]` and should pin `metadata["git_commit_id"]`.
If the old backend used `custom_tests_dir`, move those tests into each task:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
tasks/my-task/
├── instruction.md
├── environment/
│ └── Dockerfile
└── tests/
└── test.sh
```
Then pass `grader=None` to use the task-native verifier.
### 4. Package Workflow Code Instead of Mounting It
For an `AgentWorkflow`, the backend builds a wheel from `code_dir` and installs it in the container at trial start. The directory must contain `pyproject.toml` and one importable top-level Python package. When `code_dir` is omitted, the backend tries to locate the project containing the workflow class.
Use `bundle=` when your build system creates the Osmosis bundle wheel ahead of time. Do not pass both paths expecting them to be merged; a supplied bundle is used directly.
The old `HarborAgentWorkflowContext.environment` adapter is gone. The workflow itself now runs inside the task container, receives a standard `AgentWorkflowContext`, and accesses files or processes through normal Python APIs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
import asyncio
from osmosis_ai.rollout import AgentWorkflow, AgentWorkflowContext
class HarborWorkflow(AgentWorkflow):
async def run(self, ctx: AgentWorkflowContext) -> None:
result = await asyncio.create_subprocess_exec(
"python",
"/workspace/run_task.py",
)
await result.wait()
```
`OsmosisInstalledAgent` is also removed. Do not instantiate or subclass it; pass an `AgentWorkflow` through `agent=`, or select a registered native Harbor agent name.
### 5. Choose an Agent and Reward Source
`agent=` accepts an `AgentWorkflow` class/import path or one of the registered native names: `"terminus-2"`, `"mini-swe-agent"`, and `"oracle"`.
`"oracle"` runs a reference solution to validate a dataset or verifier. It does not emit a model trajectory and must not be used for training.
Choose one reward path:
| Configuration | Reward source |
| ----------------- | ------------------------------------------------------ |
| `grader=MyGrader` | The bundled Osmosis grader runs as the Harbor verifier |
| `grader=None` | The task's own `tests/` produce the reward |
### 6. Add Prewarming and Lifecycle Controls
Prewarm the selected task images and agent setup before the rollout server accepts traffic:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
from osmosis_ai.rollout.server import create_rollout_server
app = create_rollout_server(
backend=backend,
lifespan=backend.prewarm_lifespan(),
)
```
Dataset mode requires explicit task IDs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
app = create_rollout_server(
backend=backend,
lifespan=backend.prewarm_lifespan(["task-a", "task-b"]),
)
```
If you set `max_queue_depth`, `POST /rollout` returns `429` with `Retry-After: 5` when the queue is full. v0.3 also exposes:
* `GET /rollout/{rollout_id}/status`
* `POST /rollout/cancel`
* `HarborBackend.rollout_status()`
* `HarborBackend.cancel_rollouts()`
### 7. Verify a HarborBackend Migration
1. Confirm that the old class and constructor keywords no longer appear:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
rg "HarborBackendV2|HarborAgentWorkflowContext|OsmosisInstalledAgent|task_dir=|user_code_dir=|workflow=|custom_tests_dir=|prebuild_local_image=|symlink_environment=" rollouts
```
2. Confirm that `tasks_dir` points to a valid template task or dataset root.
3. Build the workflow bundle and fix any package-layout or missing-dependency errors.
4. Run `await backend.prewarm()` for template mode, or pass task IDs in dataset mode.
5. Submit one rollout and check the workflow sample, reward source, Harbor logs, and archived artifacts.
6. Exercise status and cancellation if the calling controller depends on them.
7. Run an evaluation and compare rewards and final messages with the last successful v0.2 run before training.
### Common HarborBackend Problems
The import resolves to the v0.3 backend, but the call still uses the removed constructor. Replace all stale parameters with the table above.
Point `code_dir` at the directory containing `pyproject.toml` and one importable package. Ensure the workflow and grader can be addressed as import paths from that package.
Set `metadata["harbor_task_id"]` to a directory beneath `tasks_dir`, or set `metadata["harbor_task"]` to a supported local, package, or Git reference.
Pass an Osmosis `grader`, or set `grader=None` and confirm that the selected task contains a working `tests/` verifier.
Remove the legacy adapter calls. The workflow now runs inside the container, so use normal filesystem, subprocess, and network APIs.
## Related Resources
* [Execution Backends](/cli/rollout/execution-backends)
* [Building AgentWorkflows](/cli/rollout/agent-workflows)
* [Building Graders](/cli/rollout/graders)
* [Changelog](/changelog)
# Benchmarks
Source: https://docs.osmosis.ai/platform/benchmarks
Add benchmarks, submit benchmark runs, and compare agents on the Osmosis platform
A benchmark is a published set of tasks with its own environment and grading. A benchmark run scores one or more agents on those tasks, and the platform owns the task environment, execution, and result collection, so you don't provision anything. Results land on a leaderboard that compares every agent you have run on that benchmark.
## Concepts
### Benchmark and Benchmark Run
A **benchmark** lives in your workspace and holds the task list, the harness and judge requirements, and the pass threshold. A **benchmark run** is one execution against a task selection. Runs are independent: submit as many as you need to compare agents, task subsets, or attempt budgets.
### Agent
An **agent** is a harness plus a model, for example `codex` with `openai/gpt-5.2`. One run can carry several agents, which is how you compare scaffolds or models under identical conditions. Every agent in a run gets the same tasks, the same attempt budget, and the same grading.
### Attempts and pass\@k
`attempts_per_task` sets how many independent tries each agent gets per task. Pass\@1 is the first-try rate; pass\@k is the rate of solving a task within k tries. Both are reported with a 95% confidence interval, and an agent the test can't separate from the best one shares rank 1 rather than being ranked below it. Hover its rank to see the comparison.
### Task Selection
Run every task, or narrow the selection by named task set, category, or explicit task names. Some benchmarks publish a **parity** task set, the sample their reference scores were measured on. HLE is the case to know: prefer its parity set when you want a number comparable with published results.
## Adding a Benchmark
Open **Benchmarks** in the sidebar. Osmosis-managed benchmarks are already in your workspace; **Add Benchmark** adds any dataset from the Harbor registry by name.
The table lists what your workspace can run:
| Column | Meaning |
| ------------ | -------------------------------------------------------------------------------------------------------------------- |
| **Name** | The benchmark; the row opens its page. |
| **Last Run** | The newest run's state, its age, and its name. Before the task list finishes syncing, this reports the sync instead. |
| **Tasks** | How many tasks the benchmark holds. Empty while the task list syncs, `unavailable` if the sync failed. |
| **Added** | When the benchmark entered the workspace. |
| **Added By** | Who added it. |
A Harbor benchmark's task list pages in from the registry after you add it, and runs can't be submitted until it is ready. If the sync fails, the row reports the reason and the benchmark's page offers **Retry sync**. Benchmarks you added can be removed again from the page's actions menu; managed ones cannot.
## The Benchmark Page
Each benchmark opens on its **Leaderboard** and its **Benchmark Runs** table, with **New Run** as the entry point for a submission. The header carries the benchmark's source reference (click to copy) and, once the task list is ready, a task-count badge; the actions menu offers **View source** and, after a failed sync, **Retry sync**.
### Leaderboard
One row per entrant, where an entrant is a harness-and-model pair. Each entrant is scored by its **latest** eligible run, so re-running an agent updates its standing instead of adding a row. The table ranks by Pass\@1, Pass\@k, Cost / task, Time / task, or Tokens / task. Click a metric's column header to re-rank; the ranked metric is carried in the URL. Ranks are competition ranks: tied entrants share a rank, so ranks can skip (1, 1, 3). Clicking a row opens the run behind that score.
An agent reaches the leaderboard when:
* its run finished, and the agent itself finished;
* the run covered the full task list, or a parity set the benchmark publishes for comparison;
* every task-and-attempt slot produced a result;
* the agent has a pass\@1 score.
Every eligible entrant ranks in the same list, sorted by the selected metric. Task set and benchmark version enter only the tie test: significance is checked between entrants that ran the same task set on the same resolved version, so a parity sample and a full run, or runs on different manifest versions, are never marked as tied, since they are not the same measurement.
A filtered run (a category or a handful of task names) is deliberately not ranked. It still gets a full run page, scores, and downloads.
## Submitting a Run
**New Run** opens a form with three tabs and a running summary of what you're about to submit:
* **Tasks**: every task, a named task set, categories, or explicit task names.
* **Agents**: one entry per agent, with its harness, its model, and the workspace or personal secret record holding that provider's API key. Add more entries to compare agents in one run.
* **Run settings**: attempts per task, concurrent attempts, timeout multiplier, retries, pass threshold, and the LLM judge when the benchmark scores with one.
API keys are always referenced by secret record name, never pasted into the form. Create the records first under **Secrets**; the form flags any the benchmark requires and your workspace is missing. Submission is where billing is checked, so a workspace without valid billing is told at submit rather than being locked out of the form.
Benchmark runs incur model and sandbox charges, and model spend lands on your own provider keys. Submit a one-task run first to confirm the agent and secrets work before committing to a full benchmark.
## Status Lifecycle
| Status | Description |
| ------------ | ---------------------------------------------------------- |
| **pending** | Submitted; the platform is preparing the run. |
| **queued** | Waiting on capacity to start. |
| **running** | Agents are working through their tasks; results stream in. |
| **finished** | Every expected result landed. Scores are final. |
| **failed** | The run stopped on an error. The Logs tab has the reason. |
| **stopped** | Someone stopped the run before it completed. |
## The Run Page
A run lives at `/benchmarks/runs/` and keeps a sidebar of status, progress, duration, tokens used, LLM cost, submission details, the pinned benchmark version, and its agents. Four tabs cover the run:
* **Overview**: **Agent Results** is a sortable table of the run's agents with the same metrics as the leaderboard, plus a progress column that keeps a live duration while the run is in progress. A pass\@k curve appears once agents have enough attempts, and a per-category breakdown once scored categories exist.
* **Task Results**: a searchable, filterable table of every task-and-attempt row, with the graded output, conversation, and artifacts for any result. The toolbar's Agent filter slices to one or more agents.
* **Configuration**: the resolved run configuration as TOML, including the benchmark version it was pinned to.
* **Logs**: lifecycle events from submission through cleanup.
**LLM Cost** is model spend on your own provider keys, reported by the harness. It is not billed by Osmosis.
Metrics, task-level results, and per-result artifacts all download from the run page. A pending or queued run has nothing to download yet; a running one downloads a snapshot.
## Stopping a Run
Pending, queued, and running runs can be stopped from the run page or its row in the runs table. The run moves to `stopped` once the platform finishes cleaning up its sandboxes. Results already ingested stay on the run.
## Next Steps
Submit and manage the same runs from a TOML config.
Reference for the benchmark TOML config.
Score your own rollout against a platform dataset.
Manage the secret records your agents reference.
# Create Your Own Rollout
Source: https://docs.osmosis.ai/platform/create-your-own-rollout
Use your AI coding agent to create a task-specific rollout
Use this path when you already have a task, dataset, or existing agent code and want an AI coding agent to help turn it into a runnable Osmosis rollout.
If you are new to Osmosis and want the shortest copy-paste path, start with [Run the Multiply Example](/platform/quickstart) instead.
This guide assumes you have completed [Onboarding](/platform/onboarding): your workspace repository is cloned, the CLI is installed and authenticated, and your AI coding environment is open in the workspace directory.
## What Workspace Skills Do
Platform-created workspace repositories include project-local Agent Skills under `.agents/skills/`. Agents that support the open Agent Skills format can use those skills to move through the same loop an experienced Osmosis user would follow:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
plan from dataset -> create rollout -> submit evaluation run -> debug failures -> prepare training run
```
It is not a replacement for the CLI. The agent still uses the Osmosis CLI as the source of truth for workspace checks, dataset validation, evaluation runs, and training run preflight.
## When to Use This Path
| Use this path when | Run the multiply example when |
| ----------------------------------------------------------------------- | -------------------------------------------------------- |
| You already know the task you want to train on | You want proof that the platform works end to end |
| You have sample data or a platform dataset | You do not want to design a dataset yet |
| You want the agent to create or adapt rollout code | You want to copy commands without making product choices |
| You are comfortable inspecting generated code and evaluation run output | You are still learning the Osmosis workflow |
## Use the Skills in Your Workspace Repository
Open your platform-created workspace repository in your AI coding environment. The repository includes the workspace contract and Agent Skills alongside rollout code, configs, and data:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
repository/
├── .agents/
│ └── skills/
├── .claude/
│ └── skills/
├── rollouts/
├── configs/
│ ├── eval/
│ └── training/
├── data/
├── AGENTS.md
├── CLAUDE.md
└── pyproject.toml
```
`AGENTS.md` contains the always-loaded workspace contract. `.agents/skills/` contains the canonical workflow skills, and `.claude/skills/` exposes the same skills to Claude Code through symlinks back to `.agents/skills/`.
## Start in a Workspace Repository
The skills assume this repository layout for source files:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
repository/
├── rollouts/
├── configs/
│ ├── eval/
│ └── training/
├── data/
└── pyproject.toml
```
Before asking the agent to write rollout code, confirm that the CLI can resolve the workspace:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
osmosis auth whoami
```
## Ask the Agent to Plan from the Dataset
Start by describing your task and asking the agent to begin with the workspace's planning skill. A useful first prompt is:
```text wrap theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
I want to train a model for in this Osmosis workspace. Start with the `plan-training` skill: read the workspace instructions, help me settle the dataset plan, and propose the next step before creating rollouts, running evaluation runs, or submitting a training run.
```
The workspace skills should guide the agent to:
Inspect `data/`, existing rollouts, and workspace config. The agent should settle the dataset schema before writing rollout code.
Write the smallest `AgentWorkflow` and `Grader` that can load, run, and score samples. Generated files should stay under `rollouts/`, `configs/eval/`, `configs/training/`, and `data/`.
Push the rollout to the workspace repository and submit an evaluation run as the quality gate:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git push
osmosis eval submit configs/eval/.toml
osmosis eval info
```
Fix loading, dataset, grader, dependency, and reward issues before a training run. A passing evaluation run is the handoff point from creation to training run readiness.
Once the rollout is validated, let the agent inspect the training run config and run submit-time preflight. Submit only when you are ready to start a platform training run.
Do not skip the evaluation run gate. `osmosis train submit` should be the step after the rollout cleanly loads, runs, and grades samples on the platform.
## Workspace Skills
The workspace skills are organized around rollout creation stages:
| Skill | Purpose |
| ------------------- | ----------------------------------------------------------------------------------- |
| `plan-training` | Turn a task idea or dataset into a concrete experiment plan |
| `create-rollouts` | Create or adapt rollout code, graders, entrypoints, and baseline evaluation configs |
| `evaluate-rollouts` | Submit evaluation runs, compare baselines, and inspect failures |
| `debug-rollouts` | Diagnose evaluation, config, dataset, dependency, or preflight failures |
| `submit-training` | Prepare a training run config, submit a training run, and check training run status |
You usually do not need to invoke these skills by name. Describe the outcome you want, and the agent should apply the right stage.
## Next Steps
Understand the `AgentWorkflow` and `Grader` contract behind generated rollout code.
Validate rollouts with an evaluation run before submitting a training run.
Push rollout changes and let the platform sync the code version used for evaluation runs and training runs.
Submit and monitor a training run after the evaluation run passes.
# Datasets
Source: https://docs.osmosis.ai/platform/datasets
Upload JSONL, CSV, or Parquet datasets and validate required columns for training and evaluation runs
Datasets provide the prompts and optional reference answers that drive evaluation runs and training runs. Each row becomes an example that your rollout and Grader process.
## Dataset Format
Osmosis accepts datasets in **JSONL**, **CSV**, or **Parquet** format, up to **5 GB** per file.
Each dataset must contain at least **4 rows**.
### Required Columns
| Column | Description |
| --------------- | --------------------------------------------------------- |
| `system_prompt` | The system prompt provided to the model for this example. |
| `user_prompt` | The user prompt or question the model must respond to. |
### Optional Columns
| Column | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ground_truth` | The expected correct answer or reference output. The platform UI also accepts `label` as an alias for this column. When present, the value is passed to your Grader as `context.label`. |
| `metadata` | Per-row JSON object exposed to your AgentWorkflow and Grader as `ctx.metadata`. Use it to attach context the model or grader needs (such as tags, identifiers, or expected tool calls) without baking it into the prompt. |
Include `ground_truth` (or `label`) when your Grader needs a reference answer to score against. Datasets that drive reward functions based purely on model behavior can omit it. Rows with only `metadata` (no `ground_truth`) still run through the Grader.
#### Metadata Validation Rules
`osmosis dataset upload` and `osmosis dataset validate` enforce the following rules on the `metadata` column for CSV, JSONL, and Parquet:
* Each cell must be a JSON object (a dictionary). The CLI parses CSV cells and JSONL strings as JSON, and Parquet accepts a struct column, a null column, or a JSON-object string column.
* Nested empty objects (`{}` inside the top-level object) fail validation. A top-level `{}` is fine for individual rows, but every sampled row cannot be an empty object.
* Value types for each key must stay consistent across rows. For example, `metadata.tag` cannot be a string in one row and a number in another.
* Integer values must fit in a signed 64-bit range.
* The CLI treats empty strings and missing values as absent and skips them during validation.
### Example JSONL
```jsonl theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{"system_prompt": "You are a helpful math tutor.", "user_prompt": "What is 15 * 23?", "ground_truth": "345"}
{"system_prompt": "You are a helpful math tutor.", "user_prompt": "Simplify 3/9.", "ground_truth": "1/3"}
```
## Upload a Dataset
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset upload data/train.jsonl
```
The uploaded dataset is named from the file stem (`train` in this example). After upload, the dataset enters a processing pipeline. You can check its status:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset info
```
| Status | Description |
| -------------- | -------------------------------------------------------------- |
| **uploading** | File upload has started and is not complete yet. |
| **pending** | Upload received, waiting to be processed. |
| **processing** | Dataset is being validated and indexed. |
| **uploaded** | Dataset is ready for use in evaluation runs and training runs. |
| **error** | Processing failed — check column names and file format. |
| **cancelled** | Upload was cancelled before processing completed. |
## Validate Locally
Before uploading, validate your dataset locally to catch format issues early:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset validate data/train.jsonl
```
This checks required columns, file format, and basic JSONL/CSV/Parquet structure without uploading to the platform.
## Preview a Dataset
Preview the first few rows of an uploaded dataset:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset preview my-dataset --rows 5
```
## Manage Datasets
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# List all datasets in the current workspace
osmosis dataset list
# Download a dataset file
osmosis dataset download my-dataset
```
## Next Steps
Use validated datasets in training configs.
Choose base models and deploy trained LoRA models.
# Evaluation Runs
Source: https://docs.osmosis.ai/platform/evaluation-runs
Submit, monitor, and manage evaluation runs on the Osmosis platform
An evaluation run scores your rollout's `AgentWorkflow` and `Grader` against a platform dataset and reports an aggregate score, pass rate, and per-sample results. The platform pulls your code from the synced workspace repository and runs the evaluation on its own infrastructure — you don't need GPUs or a training run.
## Concepts
### Smoke Test or Formal Evaluation
There are two reasons to run one:
* **As a smoke test before training.** Run an evaluation first to confirm the rollout works end-to-end and the grader returns reasonable scores on a small slice, before you commit GPUs to a full training run. Set a small `[evaluation].limit` to score only a few rows.
* **As a formal evaluation.** Measure agent quality on its own — to compare models or prompts, track quality over time, or run evaluations from CI. This works the same for a base model or a trained checkpoint. Set `[evaluation].limit` to the dataset's row count to score every row; otherwise the platform scores a random 10% sample.
### Evaluation Configuration vs Evaluation Run
An **Evaluation Configuration** is the recipe — it defines which model, dataset, AgentWorkflow, and evaluation settings to use. An **Evaluation Run** is a single execution of that configuration. You can submit multiple runs from the same configuration to compare models, prompts, or dataset slices.
## Submitting an Evaluation Run
Submit an evaluation run using the CLI with a TOML configuration file under `configs/eval/`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit configs/eval/my-rollout.toml
```
Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set `branch` to use a pushed branch or `commit_sha` for a specific pushed revision; omit both to use the default branch.
Pass `--yes` to skip the confirmation prompt in scripts or CI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval submit configs/eval/my-rollout.toml --yes
```
### Key Configuration Fields
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
rollout = "my-rollout" # Rollout directory name (under rollouts/)
entrypoint = "main.py" # Entrypoint file name
model_path = "openai/gpt-5-mini" # LiteLLM-style model name
dataset = "my-dataset" # Platform dataset name
# branch = "my-feature" # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..." # Optional: pin to a specific synced commit
[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200 # First N rows; omit for random 10% sample
# n = 1 # Evaluation attempts per row
# batch_size = 1 # Rows evaluated per batch
# pass_threshold = 1.0 # Minimum passing score
# agent_workflow_timeout_s = 450 # Agent workflow timeout per row
# grader_timeout_s = 150 # Grader timeout per row
```
`branch` and `commit_sha` are mutually exclusive. With `branch`, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository's default branch.
See [Config Files](/cli/config-files#eval-config) for the full TOML reference with all available fields, including `[env]` and `[secrets]`.
## Status Lifecycle
An evaluation run moves through these statuses:
| Status | Description |
| ------------ | ------------------------------------------------------------------------------------- |
| **pending** | Run is queued and waiting for resources to be provisioned. |
| **running** | Evaluation is actively executing against the dataset. |
| **finished** | Evaluation completed successfully. Score, pass rate, and sample counts are available. |
| **failed** | Evaluation encountered an error during execution. Check logs for details. |
| **stopped** | Evaluation was manually stopped by a user via the CLI or dashboard. |
## Monitoring
Track evaluation progress through the CLI or the platform dashboard.
### CLI Commands
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# List evaluation runs for the current workspace repository
osmosis eval list
osmosis eval list --all
# Show details and results for a single run
osmosis eval info my-eval-run
```
The `info` output includes the model, dataset, rollout, and timestamps, plus the aggregate score, pass rate, and total sample count once the run finishes. While a run is `pending` or `running`, results are a live snapshot. The sidebar reports progress (rows completed and percent) and duration. Dedicated **Configuration** and **Results** sections surface the entrypoint, branch, commit SHA, dataset stats, pass thresholds, pass\@k, token limits, resolved secret scopes, `[env]` keys, and the most recent platform logs.
`n` is the number of evaluation attempts per dataset row. With `limit = L` and `n = N`, the platform runs up to `L * N` total evaluations (or `sampled_rows * n` when using sampling).
### Platform Dashboard
The web dashboard at [platform.osmosis.ai](https://platform.osmosis.ai) lists evaluation runs alongside training runs, where you can filter by status, dataset, model, and rollout, and inspect per-run scores and samples.
## Managing Runs
### Stopping a Run
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval stop my-eval-run
```
This requests a stop for a pending or running evaluation. The run moves to `stopped` once the platform finishes cleanup. Pass `--yes` to skip the confirmation prompt.
## Next Steps
Reference for the evaluation TOML config.
Upload and validate datasets for evaluation runs.
Submit a training run once your evaluation results look healthy.
# Models
Source: https://docs.osmosis.ai/platform/models
Manage base models and deploy trained LoRA models for inference
Models are split into **Base Models** and **LoRA Models**. Base models are the starting point for training. LoRA models are trained checkpoints produced by training runs; deploy a LoRA model to serve it through Osmosis inference.
LoRA model lifecycle lives under the `osmosis model` command group: list, inspect, deploy, and undeploy all act on a LoRA model by name.
## Base Models
Base models are imported from [Hugging Face](https://huggingface.co) and used as the starting point for training on Osmosis.
### Supported Base Models
Osmosis currently supports:
| Model | Description |
| ------------------------ | ---------------------------------------------- |
| `Qwen/Qwen3.6-35B-A3B` | Qwen 3.6 35B with 3B active parameters (MoE) |
| `Qwen/Qwen3.5-122B-A10B` | Qwen 3.5 122B with 10B active parameters (MoE) |
The list of supported models is expanding. Check the platform dashboard or run `osmosis model list --type base` for the latest available base models.
### List Base Models
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list --type base
```
The base model list shows model name, creation date, and creator.
## LoRA Models
LoRA models are trained checkpoints produced by training runs. The Models page lists them separately from base models and shows training run, checkpoint step, training reward, creation date, and deployment status when inference deployment is available for your account.
### Inspect LoRA Models
List LoRA models:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list --type lora
```
Show details for a single LoRA model:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model info
```
Model details include the base model, training run, checkpoint step, training reward, Hugging Face export status, and deployment status when deployment info is available.
List base models and LoRA models side by side:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list
```
When deployment info is available, the LoRA section also shows the workspace's deployment-quota summary (for example, `2 of 5 inference deployments used`).
### Deploy a LoRA Model
After a training run finishes, list its LoRA models to find one to deploy:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model list --type lora
```
Deploy a LoRA model by name:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model deploy
```
Deploying an inactive LoRA model reactivates it. Deploying an already-active LoRA model is a no-op.
### Call the Inference Endpoint
Deployed LoRA models are served through the OpenAI-compatible chat completions endpoint:
```text theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
https://inference.osmosis.ai/v1/chat/completions
```
Use your Osmosis API key and the canonical `model` value from the model detail page or `osmosis model info`. The model value has the form `:`.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
curl -X POST https://inference.osmosis.ai/v1/chat/completions \
-H "Authorization: Bearer $OSMOSIS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B:my-run-step-100",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
If inference deployment is not available for your account, deployment status, deployment quota, and endpoint snippets may be hidden.
### Undeploy
To transition a LoRA model's deployment to inactive:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis model undeploy
```
The LoRA model remains in the training run history; `undeploy` only transitions the serving deployment to inactive. `undeploy` is idempotent — calling it on an already-inactive model is a no-op.
### Requirements
* Run model commands from the workspace directory so the CLI can resolve the connected workspace from Git `origin`.
* The LoRA model must belong to a training run in the same workspace.
* Inference deployment must be available for your account. Deploying models also requires workspace billing to be in good standing — for self-serve workspaces, a valid payment method on file.
* GitHub setup must be healthy before training runs can produce new LoRA models.
## Next Steps
Upload and validate datasets for evaluation runs and training runs.
Submit training runs and inspect their LoRA models.
Review model and deployment commands and options.
# Monitoring
Source: https://docs.osmosis.ai/platform/monitoring
Monitor training run status and metrics on the platform
The platform dashboard at [platform.osmosis.ai](https://platform.osmosis.ai) provides real-time visibility into your training runs.
## Training Runs
The **Training Runs** page lists each run with its name, status, dataset, base model, rollout, reward, start time, and creator. You can search and filter runs, open a run detail page, rename a run, stop an in-progress run, or delete runs that are safe to remove.
## Run Metrics
Each training run detail page includes an **Overview** tab with summary cards and metric charts. The dashboard exposes the following metrics when they are available for the run:
| Metric | Description |
| --------------------- | -------------------------------------- |
| **Duration** | Runtime for the training run. |
| **Reward** | Current training reward value. |
| **Improvement** | Change from the baseline reward. |
| **Samples** | Number of processed examples. |
| **Training Reward** | Training reward over training steps. |
| **Validation Reward** | Validation reward over training steps. |
| **Model Entropy** | Model entropy over training steps. |
| **Response Length** | Response length over training steps. |
| **Total Length** | Total length over training steps. |
| **Truncation Ratio** | Truncation ratio over training steps. |
You can refresh individual charts and download chart data as CSV from the dashboard.
## Checkpoints and Outputs
The run detail page also includes:
* **Checkpoints** — view saved checkpoints, checkpoint step, reward, deployment status, Hugging Face upload status, uploader, and checkpoint actions.
* **Configuration** — inspect the training configuration used for the run.
* **Outputs** — inspect output artifacts when they are available.
You can also check run details, checkpoints, and metrics from the CLI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train info my-run
osmosis train info my-run --output results/my-run.json
```
# Onboarding
Source: https://docs.osmosis.ai/platform/onboarding
Set up your workspace repository and choose your first training workflow
Use onboarding once before running the quickstart or building a custom rollout. By the end, you should have a platform workspace, a connected GitHub repository, a local clone, and an authenticated CLI session.
Osmosis setup starts in the [Platform](https://platform.osmosis.ai). The platform creates or provides a **workspace repository** on GitHub, and your local **workspace directory** is a clone of that repository.
## Common Setup
Complete these steps before using [Run the Multiply Example](/platform/quickstart) or [Create Your Own Rollout](/platform/create-your-own-rollout).
If you are creating a new workspace, sign in to the platform and create it there. If you were invited, accept the invitation and open the existing workspace.
Workspace owners or admins connect a GitHub account or organization, install the Osmosis GitHub App, and create the workspace repository from the platform.
Repositories created from the platform include the starter examples `multiply-local-strands`, `multiply-local-openai`, and `multiply-harbor-strands`.
Invited workspace members use the workspace repository that already exists. Do not create a second repository for the same workspace.
The platform shows clone commands once a repository is connected. Choose HTTPS, SSH, or GitHub CLI:
```cli HTTPS theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git clone https://github.com//.git
cd
```
```cli SSH theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git clone git@github.com:/.git
cd
```
```cli GitHub CLI theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
gh repo clone /
cd
```
Install the Osmosis CLI and connect it to your account:
```bash pip theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pip install osmosis-ai
```
```bash pipx theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
pipx install osmosis-ai
```
```bash uv tool theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
uv tool install osmosis-ai
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis auth login
```
Run platform commands from inside the workspace directory. The CLI reads GitHub `origin` to identify the platform workspace connected to this repository.
From the cloned workspace directory, confirm the CLI can resolve your workspace:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
osmosis auth whoami
```
## Choose Your First Workflow
Run the included `multiply-local-openai` example end to end with copy-paste commands.
Use the workspace's project-local Agent Skills to plan training, create a rollout, and validate it locally.
## Workspace Repository vs Directory
| Term | Meaning |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Workspace repository** | The GitHub repository connected to a platform workspace. The platform creates or connects it and uses it as the source of rollout code. |
| **Workspace directory** | Your local clone of the workspace repository. Run CLI commands from here so Osmosis can infer the connected workspace from Git `origin`. |
## If Setup Is Blocked
The platform may show a GitHub banner when the workspace is missing required repository setup.
| State | What to do |
| --------------------------- | ----------------------------------------------------------------------------- |
| No GitHub account connected | Workspace owners or admins should connect GitHub from **Git Integration**. |
| No repository connected | Workspace owners or admins should create or connect the workspace repository. |
| GitHub App disconnected | Reconnect the GitHub App before submitting a training run. |
| Repository needs setup | Open **Git Integration** and follow the repository repair flow. |
For details on how Git Sync controls the code version used for training, see [Git Sync](/cli/workspace/git-sync).
## Next Steps
Copy and paste the shortest path from cloned repository to training run.
Build a task-specific rollout with project-local Agent Skills and evaluation run gates.
Understand how the CLI maps your Git clone to a platform workspace.
# Overview
Source: https://docs.osmosis.ai/platform/overview
Understand the Osmosis web dashboard for managing training runs, datasets, models, and more
[Osmosis Platform](https://platform.osmosis.ai) is the web dashboard for managing reinforcement learning training of LLMs. It handles workspace setup, GitHub repository connection, GPU provisioning, training orchestration, metrics collection, and checkpoint deployment so you can focus on defining agent behavior and evaluation logic.
## Core Capabilities
Organize your team, datasets, training runs, models, and workspace repository access with role-based permissions.
Submit, monitor, and manage RL training runs with configurable hyperparameters and checkpoint cadence.
Upload and validate JSONL, CSV, or Parquet datasets up to 5 GB for training.
List supported base models and deploy trained LoRA models for inference.
Track training run status, metrics, checkpoints, and outputs.
Create or connect a workspace repository to sync rollouts and configs automatically.
## How It Works
The typical workflow from setup to deployed LoRA model follows five stages:
Start with [Onboarding](/platform/onboarding) to create or join a platform workspace, connect GitHub, clone the workspace repository, install the CLI, and verify local workspace context.
Run the included [Multiply example](/platform/quickstart) for a known-good first training run, or use [Create Your Own Rollout](/platform/create-your-own-rollout) when you already have a task or dataset.
Push rollout changes to GitHub. Git Sync publishes the code version, then `osmosis eval submit` starts an evaluation run against a platform dataset to catch dataset, dependency, workflow, and grader issues before a training run.
Once evaluation run results look healthy, `osmosis train submit` starts the training run.
Track metrics, checkpoints, and outputs in the dashboard. When a run finishes, deploy a LoRA model.
## Ready to Get Started?
Follow the full workspace setup flow for creators and invited members.
After onboarding, run the included example from evaluation run to training run.
# Run the Multiply Example
Source: https://docs.osmosis.ai/platform/quickstart
Run your first RL training loop with a starter Multiply example
This guide is the fastest post-onboarding path: copy the commands, run the included `multiply-local-openai` example, and submit one training run before customizing anything.
If you already have a dataset or task-specific rollout in mind, use [Create Your Own Rollout](/platform/create-your-own-rollout) instead.
This guide assumes you have completed [Onboarding](/platform/onboarding): your workspace repository is cloned, the CLI is installed and authenticated, and commands are running from the workspace directory.
Confirm that the CLI can resolve your workspace before running the example:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis doctor
```
Evaluation runs and training runs both reference platform datasets by name. Upload the dataset that ships with the starter example — both `configs/eval/multiply-local-openai.toml` and `configs/training/multiply-local-openai.toml` reference it as `multiply`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis dataset upload data/multiply.jsonl
```
The starter evaluation run calls the OpenAI endpoint, so `[secrets]` in `configs/eval/multiply-local-openai.toml` maps `OPENAI_API_KEY` to a workspace secret record. Register that secret at `/:orgName/secrets` in the platform UI before submitting, so the platform can inject the key into the evaluation run container.
Push the repository so the platform can clone the rollout code, then submit an evaluation run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
git push
osmosis eval submit configs/eval/multiply-local-openai.toml
```
Inspect progress and results:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis eval list
osmosis eval info
```
Once the evaluation run looks healthy, submit a training run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train submit configs/training/multiply-local-openai.toml
```
Monitor the run and deploy a LoRA model when it finishes:
```cli theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train info
osmosis model deploy
```
At this point, you have run the full Osmosis loop with known-good code: dataset upload, evaluation run, training run submission, and deployment inspection. Use the rollout docs only after this works.
## Next Steps
Use project-local Agent Skills when you are ready to adapt Osmosis to your own task.
Learn about training configuration, statuses, and management.
Understand dataset formats and validation.
Manage base models and deploy trained LoRA models.
# Settings
Source: https://docs.osmosis.ai/platform/settings
Manage workspace settings, members, secrets, integrations, and billing
## Workspace Settings
Workspace settings are available from **Workspace Settings** in the platform sidebar. Available pages depend on your role.
### General
The **General** page shows the workspace ID and lets owners and admins update:
* **Workspace Name** — the workspace URL identifier.
* **Logo** — the workspace logo image.
* **Default Timezone** — the timezone used when displaying dates and times in the workspace.
### Members
The **Members** page lists workspace members by name, role, and join time.
| Role | Permissions |
| ---------- | --------------------------------------------------------------------- |
| **Owner** | Full access, including workspace deletion and managing all roles. |
| **Admin** | Manage settings, billing, integrations, members, and API keys. |
| **Member** | View and manage resources like datasets, training runs, and rollouts. |
Owners and admins can invite members by email, assign a role, and view pending invites.
### Webhooks
The **Webhooks** page lets owners and admins configure a URL that Osmosis notifies when training and evaluation runs finish. See [Webhooks](/platform/webhooks) for setup and the payload reference.
### Danger Zone
The **Danger Zone** page lets owners manage workspace deletion. Non-owner members see the option to leave the workspace.
***
## Secrets
Secrets are available from **Secrets** in the platform sidebar. Use secrets to store environment values that are needed during training.
The **Secrets** page lists each secret by name, masked value, creator, and creation time. Secret values are encrypted at rest, masked by default in the UI, and can be revealed or copied from the table.
***
## Integrations
Integrations are available from **Integrations** in the platform sidebar.
### Git
The **Git Integration** page connects GitHub to manage rollouts, training configs, and related workspace content. Owners and admins can connect or disconnect the GitHub account and manage the connected repository.
See [Git Sync](/cli/workspace/git-sync) for the repository workflow.
### Hugging Face
The **Hugging Face Integration** page manages Hugging Face tokens for uploading checkpoints to Hugging Face. A workspace can store up to 20 Hugging Face tokens.
***
## Billing
Usage and pricing are visible to all workspace members. Invoices and payment
settings are only available to workspace owners and admins.
Osmosis uses pay-as-you-go billing: resource usage (GPU training time, sandboxes, and rollout servers) is metered per run and invoiced monthly. Paid features require a payment method on file, and adding your first payment method grants a one-time \$100 welcome credit.
The **Billing** page has four tabs:
* **Usage** — total cost and a daily cost chart for the selected time range, plus a per-run usage table. Owners and admins also see the remaining credit balance. Use the range selector to switch between the current or previous billing cycle, recent rolling windows (last 7, 14, or 30 days, or last 3 months), or all time.
* **Invoices** — monthly invoices with status and totals; download individual PDFs, or select several to download as a ZIP.
* **Payment** — manage payment methods (add a card, choose the default) and billing information such as the billing email and business address. Invoiced workspaces also see their bank transfer details here.
* **Pricing** — current resource rates, shown per hour or per second.
### Billing Modes
* **Self-serve** (default) — the previous month's usage is charged automatically to the default payment method at the start of each month.
* **Invoiced** — for teams that pay by bank transfer (ACH or wire), Osmosis issues monthly invoices due within 30 days instead of charging a card. [Contact us](mailto:founders@osmosis.ai) to switch billing modes in either direction.
# Training Runs
Source: https://docs.osmosis.ai/platform/training-runs
Submit, monitor, and manage training runs on the Osmosis platform
A training run takes a base model and improves it through reinforcement learning. You provide the rollout, grader, training config, and dataset; the platform provisions GPUs, pulls code from your synced workspace repository, executes the training loop, and saves checkpoints automatically.
## Concepts
### Training Configuration vs Training Run
A **Training Configuration** is the recipe — it defines which model, dataset, AgentWorkflow, and hyperparameters to use. A **Training Run** is a single execution of that configuration. You can submit multiple runs from the same configuration to experiment with different settings.
### Training Behavior
Each submitted run is a single managed training job for the rollout, dataset, model, and hyperparameters in its TOML config. To run another experiment, submit the config again with updated fields such as `total_epochs`, sampling settings, or checkpoint cadence.
## Submitting a Training Run
Submit a training run using the CLI with a TOML configuration file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train submit configs/training/default.toml
```
Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set `branch` to use a pushed branch or `commit_sha` for a specific pushed revision; omit both to use the default branch.
### Key Configuration Fields
```toml theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
[experiment]
rollout = "my-rollout" # Rollout directory name (under rollouts/)
entrypoint = "main.py" # Entrypoint file name
model_path = "Qwen/Qwen3.6-35B-A3B" # Hugging Face model path
dataset = "my-dataset" # Dataset name
# branch = "my-feature" # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..." # Optional: pin to a specific synced commit
[training]
lr = 1e-6 # Learning rate
total_epochs = 1 # Number of training epochs
n_samples_per_prompt = 8 # Samples generated per prompt
rollout_batch_size = 32 # Rollout batch size
agent_workflow_timeout_s = 450 # Agent rollout timeout per row
grader_timeout_s = 150 # Grader timeout per row
[sampling]
rollout_temperature = 1.0 # Sampling temperature during rollouts
rollout_top_p = 1.0 # Top-p sampling during rollouts
[checkpoints]
checkpoint_save_freq = 20 # Save checkpoint every N steps
```
`branch` and `commit_sha` are mutually exclusive. With `branch`, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository's default branch.
See [Config Files](/cli/config-files) for the full TOML reference with all available fields.
## Status Lifecycle
Every training run progresses through a series of statuses:
| Status | Description |
| ------------ | ----------------------------------------------------------------------------- |
| **pending** | Run is queued and waiting for GPU resources to be provisioned. |
| **running** | Training is actively in progress. Metrics and checkpoints are being produced. |
| **finished** | Training completed successfully. Final checkpoint and metrics are available. |
| **failed** | Training encountered an error during execution. Check logs for details. |
| **stopped** | Training was manually stopped by a user via the CLI or dashboard. |
| **killed** | Training was terminated during platform cleanup or stop handling. |
| **crashed** | Training process terminated unexpectedly (e.g. OOM, hardware failure). |
| **unknown** | The platform could not determine the current training state. |
The internal lifecycle phases are: **init** → **provision** → **setup** → **train** → **finalize** → **complete** (or **error** / **cleanup**).
A run in `failed` or `crashed` status may still have usable checkpoints saved before the failure occurred.
## Monitoring
Track training progress through the CLI or the platform dashboard.
### CLI Commands
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
# Show run details, checkpoints, and metrics
osmosis train info my-run
# Save metrics to a specific JSON file
osmosis train info my-run --output results/my-run.json
```
While a run is in flight, `train info` reports progress (`current_step` / `total_steps`) and the most recent reward. `train list` surfaces the same fields so you can scan runs at a glance.
### Platform Dashboard
The web dashboard at [platform.osmosis.ai](https://platform.osmosis.ai) provides:
* **Run list** — search and filter runs by status, dataset, base model, and rollout.
* **Overview metrics** — view Duration, Reward, Improvement, Samples, Training Reward, Validation Reward, Model Entropy, Response Length, Total Length, and Truncation Ratio when available.
* **Checkpoints** — view saved checkpoints with their step, reward, deployment status, and Hugging Face upload status.
* **Outputs** — inspect output artifacts when they are available.
See [Monitoring](/platform/monitoring) for the full list of dashboard metrics.
## LoRA Checkpoints
During training, LoRA checkpoints are saved at the interval specified by `checkpoint_save_freq` in your configuration. Checkpoints capture the adapter weights at a specific training step.
You can:
* **Compare checkpoints** by their reward scores to find the best-performing step
* **Export checkpoints** from the dashboard
* **Upload checkpoints** to Hugging Face
* **Deploy LoRA models** for inference with `osmosis model deploy`
## Managing Runs
### Stopping a Run
```bash theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
osmosis train stop my-run
```
This requests a graceful stop for the training process. If the stop completes successfully, the run enters `stopped` status.
## Next Steps
Upload datasets for training.
Manage base models and deploy trained LoRA models.
# Webhooks
Source: https://docs.osmosis.ai/platform/webhooks
Receive an HTTP POST when a training or evaluation run finishes
Webhooks notify your systems when runs finish. Osmosis sends a `POST` request with a JSON payload to a URL you configure, once per completed run.
## Setup
Open **Workspace Settings → Webhooks** (owners and admins only):
1. Enter your **Webhook URL**. It must use HTTPS on port 443 and resolve to a public address. URLs that point at private or internal networks are rejected.
2. Turn on **Deliver events** and click **Save**.
3. Click **Send test payload** to confirm your receiver gets a `webhook.test` event.
Turn off **Deliver events** to pause deliveries without losing the URL. To remove the webhook entirely, clear the **Webhook URL** and click **Save**.
## Events
| Event | Sent when |
| ------------------------ | ------------------------------- |
| `training_run.completed` | A training run finishes |
| `eval_run.completed` | An evaluation run finishes |
| `webhook.test` | You click **Send test payload** |
## Payload
Every request is a `POST` with `Content-Type: application/json` and this envelope:
```json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{
"event": "training_run.completed",
"timestamp": "2026-07-15T22:14:05.123456+00:00",
"data": {}
}
```
### `training_run.completed` and `eval_run.completed`
The `data` object has the same shape for both events:
| Field | Description |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | The run ID |
| `name` | The run name |
| `status` | Final status of the run: `finished`, `failed`, or `stopped` |
| `started_at` | When the run started, ISO 8601 |
| `completed_at` | When the run finished, ISO 8601 |
| `duration_seconds` | Total run duration in seconds |
| `model_name` | Name of the model used by the run |
| `dataset` | `{ "id", "name" }` of the dataset, or `null` |
| `rollout` | `{ "id", "name" }` of the rollout, or `null` |
| `platform_url` | Link to the run detail page in the platform |
| `latest_metrics` | For training runs, the latest logged value of each run metric keyed by metric name. For evaluation runs, the run's final aggregate results, including the reward distribution (`reward_stats`) |
Training run example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
{
"event": "training_run.completed",
"timestamp": "2026-07-15T22:14:05.123456+00:00",
"data": {
"id": "a1b2c3d4-0000-0000-0000-000000000000",
"name": "my-training-run",
"status": "finished",
"started_at": "2026-07-15T20:01:12+00:00",
"completed_at": "2026-07-15T22:14:03+00:00",
"duration_seconds": 7971,
"model_name": "Qwen3-8B",
"dataset": { "id": "d1e2f3a4-0000-0000-0000-000000000000", "name": "my-dataset" },
"rollout": { "id": "r1s2t3u4-0000-0000-0000-000000000000", "name": "my-rollout" },
"platform_url": "https://platform.osmosis.ai/my-workspace/training/a1b2c3d4-0000-0000-0000-000000000000",
"latest_metrics": {
"rollout/raw_reward": 0.82,
"eval/validation/reward": 0.79,
"train/entropy_loss": 1.24,
"rollout/response_lengths": 512,
"rollout/total_lengths": 1930,
"rollout/truncated_ratio": 0.03
}
}
}
```
For evaluation runs, `latest_metrics` contains the final aggregate results instead:
```json theme={"theme":{"light":"github-light","dark":"github-dark"},"languages":{"custom":["/languages/cli.json"]}}
"latest_metrics": {
"total_runs": 200,
"graded": 198,
"passed": 154,
"failed": 44,
"skipped": 2,
"pass_rate": 0.78,
"pass_threshold": 0.5,
"score": 0.71,
"reward_stats": {
"mean": 0.71,
"median": 0.74,
"std": 0.18,
"min": 0.05,
"max": 0.98,
"pass_at_k": { "1": 0.78, "4": 0.91 }
},
"total_tokens": 1834520,
"total_duration_ms": 5421000
}
```
### `webhook.test`
| Field | Description |
| ----------- | ---------------------- |
| `workspace` | The workspace name |
| `message` | A confirmation message |
## Delivery
* Each attempt times out after 5 seconds. Failed deliveries are retried up to 2 times with backoff.
* Any 2xx response counts as delivered. Redirects are not followed.
* Delivery is best effort: a webhook failure never affects the run itself.