Osmosis CLI Command Reference

Run osmosis -h to see all available commands. Every sub-command supports -h / --help. The global output flags --json and --plain are position-independent — they work before or after the command:

osmosis --json <command>
osmosis <command> --json

Use --json for automation and AI agents. Use --plain for low-noise shell output.

Workspace Scope

By default, workspace-scoped Platform commands derive their scope from the current repository’s Git origin. Root option --workspace <workspace-name> selects an exact platform workspace by name instead and sends only X-Osmosis-Workspace, never X-Osmosis-Git alongside it:

osmosis --workspace <workspace-name> --json dataset list
osmosis --workspace <workspace-name> benchmark runs info <run-name>

With explicit scope, no local repository is required for the complete benchmark catalog, submit, and run family; platform dataset, model, and secret commands; or train list, info, logs, and stop; eval list, info, logs, retry, and stop. Without --workspace, these commands retain Git-derived scope. eval run, eval upload, eval download, and the rollout commands are not in that set: each reads or writes a workspace-relative layout on disk, so they still resolve the workspace from the current repository even when --workspace is set. Source-backed eval submit and train submit also accept root --workspace from any current directory when config_path is absolute. The config must still live under the containing Osmosis Git workspace’s canonical configs/eval/ or configs/training/ directory. The CLI locates that repository, verifies the selected workspace is connected to it, and submits using only workspace-name scope. Structured results under explicit platform-only scope include workspace.name and omit fabricated git and workspace_directory fields. Source-backed eval and training submit may additionally include the real local Git context resolved from the config path.

Environment Variables

The CLI loads the nearest .env file, walking upward from the current directory, before it reads these variables. Non-empty process variables win over dotenv values. Use root option --env-file <path> (or OSMOSIS_ENV_FILE) to select a file, and --platform <url> to override the active platform for one command.

Variable	Purpose
`OSMOSIS_TOKEN`	Authenticate non-interactively instead of using stored credentials; `osmosis auth login` is disabled while it is set
`OSMOSIS_TOKEN_PLATFORM_URL`	Bind `OSMOSIS_TOKEN` to its intended platform; required when the active platform is not the default production platform
`OSMOSIS_TOKEN_STORE`	Select storage for new persistent logins: `auto` (default; keyring with file fallback), `keyring` (require the keyring), or `file` (save to `~/.config/osmosis/credentials.json`)
`OSMOSIS_PLATFORM_URL`	Override the Platform base URL; scheme-less loopback hosts use HTTP and other hosts use HTTPS
`OSMOSIS_ENV_FILE`	Load a specific dotenv file instead of discovering the nearest `.env`
`OSMOSIS_ALLOW_INSECURE_PLATFORM_URL`	Set to `1` to allow a non-HTTPS, non-loopback platform URL that would otherwise fail closed
`OSMOSIS_DEBUG`	Set to `1` to append the original exception and traceback after an unexpected `INTERNAL` error without changing its JSON envelope

export OSMOSIS_TOKEN="<token>"
export OSMOSIS_PLATFORM_URL="<non-production-platform-url>"
export OSMOSIS_TOKEN_PLATFORM_URL="$OSMOSIS_PLATFORM_URL"
osmosis --json auth whoami

quickstart

Interactive setup: sign in, clone your workspace, and get your first agent prompt.

osmosis quickstart [--workspace NAME]

Option	Type	Default	Description
`-w`, `--workspace`	`str`	Current clone, otherwise prompt	Workspace name to set up

The wizard authenticates the CLI, resolves which workspace to use, waits for the workspace repository to be connected and clones it, reports whether billing is set up, then asks what you want to do. Training a model, evaluating a model on a dataset, and running a benchmark each print an agent prompt for that goal; exploring prints documentation links instead. When you run osmosis quickstart from inside a workspace clone, the wizard picks up that workspace from the git remote. From an empty or unrelated directory, it lists the workspaces you belong to and asks you to pick one. Pass --workspace <name> to skip the picker and target a specific workspace by name. This is useful when you belong to more than one workspace. It is safe to re-run: every step checks the current state before changing anything, so an already-configured workspace gets an all-clear report and goes straight to asking what you want to do next. Re-running from an empty directory reuses the last clone path recorded for that workspace when it still exists. The wizard needs an interactive terminal and runs in rich output mode only, so it is not available with --json or --plain — see Onboarding for the manual steps it automates.

auth

Manage CLI authentication.

Command	Description
`osmosis auth login`	Authenticate via browser OAuth or personal access token
`osmosis auth logout`	Revoke the current session and clear local credentials
`osmosis auth whoami`	Show the authenticated user, token expiration, and linked workspace

osmosis auth login
osmosis auth whoami

When you run osmosis auth whoami from inside a workspace repository, the CLI also resolves the linked Platform workspace. --json reports it as a workspace object (id, name, role); other modes show Workspace and Role rows. Outside a workspace repository, or when the repository is not linked to a Platform workspace, workspace is null and the rest of the output is unchanged. The command always verifies the stored credential or OSMOSIS_TOKEN against the Platform, so it needs network access and it fails with AUTH_REQUIRED when you are logged out or your session has expired. Persistent credentials are keyed by the normalized platform URL, so logins for multiple platform environments can coexist. New logins prefer the operating-system keyring and fall back to an owner-only ~/.config/osmosis/credentials.json with a KEYRING_UNAVAILABLE warning when the keyring cannot be used. Set OSMOSIS_TOKEN_STORE=keyring to require keyring storage or OSMOSIS_TOKEN_STORE=file to save new logins to the file. Existing credentials retain their recorded backend. Run osmosis auth whoami to see the effective credential source and the backend of any persistent login. An explicit osmosis auth login replacement or osmosis auth logout changes stored credentials; an HTTP 401 reports an expired or revoked session without deleting local state. When OSMOSIS_TOKEN targets a non-production platform, bind it with OSMOSIS_TOKEN_PLATFORM_URL. A missing or mismatched binding fails before any network request with ENV_TOKEN_PLATFORM_REQUIRED or ENV_TOKEN_PLATFORM_MISMATCH. See Installation & Authentication.

doctor

Inspect and optionally repair the workspace directory scaffold.

osmosis doctor [path] [--fix]

Argument / Option	Type	Default	Description
`path`	`path`	`.`	Path inside the workspace directory
`--fix`	flag	—	Create missing scaffold directories without overwriting existing files

When you are logged in and the workspace directory has a Platform-connected origin remote, doctor also reports the linked workspace (Linked workspace: <name>, or the workspace resource field in --json). The lookup is best-effort — offline or logged out, the field is null and doctor still works.

template

Add starter rollouts from the Osmosis workspace template catalog.

template list

osmosis template list

Lists available starter templates.

template apply

osmosis template apply <name> [--force]

Argument / Option	Type	Description
`name`	`str` (required)	Template name from `osmosis template list`
`-f`, `--force`	flag	Replace the template-owned rollout directory and config files

rollout

Create local rollout scaffolds and list synced rollouts.

rollout init

osmosis rollout init <name> [--force]

Creates:

rollouts/<name>/main.py
rollouts/<name>/pyproject.toml
rollouts/<name>/README.md
configs/eval/<name>.toml
configs/training/<name>.toml

Argument / Option	Type	Description
`name`	`str` (required)	Lowercase rollout name that starts with a letter and then uses letters, digits, or hyphens. `default` is reserved.
`-f`, `--force`	flag	Replace existing scaffold paths for this rollout

rollout list

osmosis rollout list [--limit N] [--all] [--branch NAME]

Option	Type	Default	Description
`--limit`	`int`	`50`	Maximum number of rollouts to show
`--all`	flag	—	Show all rollouts
`--branch`	`str`	Repository default	Show rollouts synced from this branch

eval

Run evaluations locally, publish completed local results, submit and manage platform evaluation runs, and run LLM-as-judge rubric scoring locally.

eval run

Execute an evaluation locally against the current workspace’s rollout server through the backend its entrypoint constructs, LocalBackend or Harbor. This requires a valid local workspace and python -m pip install "osmosis-ai[eval]>=0.3.4,<0.4". With --dataset-file PATH and no --upload, the command does not require platform credentials. Selecting the platform dataset named in the config or adding --upload still requires authenticated Git workspace context. Provider credentials resolve from the process environment or --secrets-file; --secrets-file values take precedence for this run only and the original environment is restored on exit.

osmosis eval run <config_path> [--name NAME] [--output PATH] [--dataset-file PATH] [--secrets-file PATH] [--rows ROWS] [--fresh] [--retry-failed] [--max-in-flight N] [--yes] [--tunnel PROVIDER] [--rollout-port PORT] [--listener-port PORT] [--advertise-url URL] [--verbose] [--upload]

Argument / Option	Type	Description
`config_path`	`path` (required)	Eval TOML under `configs/eval/`
`--name`	`str`	Previous run name to resume; omit it for a generated `adjective-animal-number` name
`-o`, `--output`	`path`	Run output root; defaults to `.osmosis/evals/`
`--dataset-file`	`path`	Local dataset file to use instead of the platform dataset
`--secrets-file`	`path`	Dotenv file supplying `[secrets].required` values; `-` reads stdin
`--rows`	`str`	Dataset rows such as `3,7,10-20`; overrides `limit`
`--fresh`	flag	Archive existing results for this run name and start clean
`--retry-failed`	flag	Re-run failed and skipped work items while keeping successes
`--max-in-flight`	`int`	Concurrent rollouts; defaults to `evaluation.batch_size`, then backend capacity, then `1`
`-y`, `--yes`	flag	Skip the local-run confirmation prompt
`--tunnel`	`str`	Force a managed tunnel for the model bridge instead of relying on automatic detection; `cloudflared` is the only provider. Needed only when the rollout server does not report the requirement itself, such as a third-party backend
`--rollout-port`	`int`	Fixed rollout-server port; defaults to an ephemeral port
`--listener-port`	`int`	Fixed local port for the model-bridge listener; defaults to an ephemeral port
`--advertise-url`	`str`	Public base URL of a tunnel you run that already reaches the listener; requires `--listener-port` and cannot be combined with `--tunnel`
`--verbose`	flag	Stream supervisor and rollout-server logs
`--upload`	flag	Publish after the run reaches a complete terminal state

Failed and skipped samples are terminal and uploadable. Pending and cancelled runs are not. --upload runs under the same local run lock and starts publishing only after completion. Output lands under .osmosis/evals/<run-name>/ with the same metrics.json, summary.jsonl, trajectories/, and artifacts/ shape as eval download, plus the canonical rollout_trials/<rollout-id>/ store and a logs.txt that redacts configured [secrets] values and known provider and platform key values of at least 8 characters from the process environment. Re-running the same --name resumes work without a durable terminal result. If the named run is complete, an interactive command offers a new generated-name run; declining, --yes, and non-interactive modes leave it untouched and print next steps. A named run is locked to its model, dataset bytes, selected rows, n, timeouts, pass threshold, entrypoint, and rollout source digest. Changed inputs refuse resume. Use --fresh to archive the old directory and restart — previous results move to .osmosis/evals/<run-name>.archive-<utc-timestamp>/ and are never deleted — or --retry-failed without a code change to retry failed and skipped items while keeping successes. Resume is journal-backed. Terminal results returned by long polling are appended to events.jsonl and fsync-ed before they count as completed local work. After interruption, work without a durable terminal record remains pending and can run again. The rollout server runs through uv in the environment resolved from rollouts/<name>/pyproject.toml, not the CLI environment. Upgrade both the CLI and that rollout dependency to 0.3.3 or later in the 0.3.x series; the CLI and server must use the long-polling protocol introduced in 0.3.3. Put Harbor and other rollout-side dependencies in that file. A ROLLOUT_SDK_VERSION_MISMATCH warning reports different installed versions but does not establish protocol compatibility. Local runs created with the earlier callback protocol cannot resume under 0.3.3: use a new name or --fresh to archive and restart them. See the 0.3.3 migration guide. Harbor sandboxes that cannot reach this machine’s loopback interface — Daytona and Docker outside macOS — need a public URL for the local model bridge. The CLI reads that requirement from the rollout server’s health report and starts a managed cloudflared quick tunnel automatically, so keep that binary on PATH. A rollout server that does not report the requirement — such as a third-party backend — needs --tunnel cloudflared to force one, or --advertise-url with --listener-port to use a tunnel you run yourself. Startup validates the model and the rollout server before opening a tunnel, and the run stops with an error instead of hanging when the tunnel never becomes reachable or exits mid-run. Only the sandbox’s OpenAI-compatible chat traffic crosses the tunnel. Rollout results are collected by long polling over loopback. The model-bridge listener exposes no docs or OpenAPI surface; its chat routes require a fresh bearer token generated for each run, and the bridge supplies provider credentials upstream. A bare tunnel URL answers 404, which is also how startup probes the tunnel for readiness; when that probe cannot complete, the run warns and proceeds on cloudflared’s own connection-registration signal, since the sandbox may resolve the host when you cannot. A Harbor task running allowlist egress gets the tunnel host added to its allowlist automatically. Tunnel mode arms a keepalive on the bridge’s non-streaming path, because proxy edges cut silent origins — a cloudflared quick tunnel allows roughly 125 seconds between reads. After a 90-second grace window the bridge commits 200 application/json and emits one whitespace byte every 30 seconds until the real body arrives, so a slow model call is not severed mid-flight. The trade-off is that a provider error raised after that window arrives as an OpenAI-style {"error": ...} body under a 200 status instead of a non-2xx response. Cloudflare rate-limits quick-tunnel creation per source IP rather than tunnel traffic, so many short runs from one machine can start failing with 429 while already-running tunnels keep serving. Wait a few minutes, or pass --advertise-url with a tunnel you run yourself. See Cloudflare’s quick tunnel documentation for the current limits.

There is no local-only config section, so the same evaluation TOML works with eval run and eval submit. A Harbor rollout’s environment selection lives in its entrypoint and applies unchanged to local eval.

osmosis eval run configs/eval/my-rollout.toml
osmosis eval run configs/eval/my-rollout.toml --upload

eval upload

Publish an already-completed compatible local evaluation run. This command has no confirmation prompt.

osmosis eval upload <eval-name-or-directory> [--replace]

Argument / Option	Type	Description
`eval_name_or_directory`	`path` (required)	Completed local run name, resolved under the workspace’s `.osmosis/evals/`, or an explicit run directory such as `.osmosis/evals/<run-name>/`
`--replace`	flag	Replace the results this run already published, for a run re-run with `--retry-failed`

A bare run name resolves to .osmosis/evals/<run-name>/ in the workspace directory; an existing directory of that name in the current directory wins, and any relative or absolute path with a separator is used as given. The directory must contain compatible manifest.json, index.jsonl, progress.json, and metrics.json files, and the command requires authenticated workspace context. Upload is idempotent and server-authoritative: re-run the same command after interruption to return the same platform run and upload only files still missing on the server. The upload sends index.jsonl, progress.json, the combined logs.txt when present, and only referenced canonical trajectory*.json files and safe artifacts for selected rollout IDs. It does not send local manifest.json bytes, events.jsonl, metrics.json, summary or projection copies, control files, per-trial logs, or superseded attempts. Before hashing logs.txt, upload re-applies redaction for known provider and platform key values of at least 8 characters in the current process environment. It cannot detect arbitrary secrets or old credentials absent from that environment; see log redaction. Manifest digest, schema versions, and allowlisted redacted Git provenance are metadata; the server strictly validates files, recomputes metrics, and does not launch hosted or Temporal evaluation work. Resuming an interrupted upload never needs --replace. It returns the same platform run and sends only the files still missing. --replace publishes a different set of results for a run that already finished importing. Without it, that upload returns a conflict rather than overwriting the results. Pass --replace after a --retry-failed attempt to replace that run’s published results in place. The platform run keeps its ID and URL, and the manifest digest still has to match, which proves both attempts evaluated the same resolved inputs. eval run --retry-failed --upload sets it for you.

osmosis eval upload <run-name>
osmosis eval upload .osmosis/evals/<run-name>/
osmosis eval upload <run-name> --replace

eval submit

Create and run an evaluation on managed infrastructure from a TOML config under the source repository’s configs/eval/. Normally the platform clones the workspace repository identified by the current Git origin. With root --workspace, pass an absolute canonical config path; the CLI verifies its repository matches the selected workspace. Push your commits before submitting.

osmosis eval submit <config_path> [--yes] [--secrets-file PATH]

Argument / Option	Type	Description
`config_path`	`path` (required)	Eval TOML under the source repository’s `configs/eval/`; must be absolute when root `--workspace` is set
`-y`, `--yes`	flag	Skip the confirmation prompt
`--secrets-file`	`path`	Dotenv file supplying values for `[secrets].required` names; `-` reads stdin

osmosis eval submit configs/eval/my-rollout.toml

# Submit the same source-backed config from another current directory
osmosis --workspace my-workspace eval submit /absolute/path/to/repository/configs/eval/my-rollout.toml

# Supply per-run secret values without saving them to the store
osmosis eval submit configs/eval/my-rollout.toml --secrets-file .env.run
vault read -field=env secret/eval | osmosis eval submit configs/eval/my-rollout.toml --secrets-file - --yes

Each name in [secrets].required resolves in this order: --secrets-file, the CLI process environment, an existing personal or workspace secret, then a hidden prompt when stdin is a TTY. Existing stored values are resolved server-side. Values supplied locally by file, environment, or prompt are sent with this submission’s TLS request but are not added to the Osmosis secret store or persistent run config, so supply them again for later runs. Outside a TTY, names still unresolved after the stored-secret check are reported all at once so CI shows the full gap. See [env] and [secrets] for the config-side rules.

Config values come from the local TOML file. Rollout code comes from the synced workspace repository. Use [experiment].branch or [experiment].commit_sha to select the source; they are mutually exclusive.

Uploaded local runs also appear in eval list, eval info, and Platform viewers with a Local badge; dirty Git provenance produces a warning. An uploaded local run does not replace the managed full-size evaluation gate required before training.

eval list

List evaluation runs for the selected workspace, including each run’s status, average reward, and progress.

osmosis eval list [--limit N] [--all]

Option	Type	Default	Description
`--limit`	`int`	`50`	Maximum number of evaluation runs to show
`--all`	flag	—	Show all evaluation runs

eval info

Show details, results, and metrics for an evaluation run.

osmosis eval info <run-name> [--output PATH]

Argument / Option	Type	Description
`name_or_id`	`str` (required)	Evaluation run name or ID
`-o`, `--output`	`path`	Run output root. Defaults to `.osmosis/evals/<name>/` in rich mode.

The detail fields show name, status, progress (rows completed and percent), duration, pass rate, tokens used, submission and completion times, and the dataset, model, and rollout the run used. Two sections follow:

Configuration — entrypoint, the resolved config values (limit, n, batch_size, pass_threshold, timeouts), branch, short commit SHA, resolved secret scopes, and [env] keys.
Results — graded, passed, failed, and skipped counts, pass threshold, reward stats (mean, std, min, median, max), and pass@k.

eval info reports aggregates only. Use eval logs for platform logs and eval download for row-level trajectories, artifacts, metrics, and logs. For completed runs, eval info also fetches metrics (duration, pass rate, tokens used, reward statistics, and pass@k) from the platform and exports them as JSON:

In rich mode, the CLI saves metrics to .osmosis/evals/<name>/metrics.json by default when metrics are available.
In JSON or plain mode, pass -o / --output to set the run output root; the CLI writes metrics.json inside that directory.
The CLI creates parent directories automatically.
If metrics are not yet available (for example, the run is still pending), the CLI prints a notice instead of writing a file.

osmosis eval info my-eval-run
osmosis --json eval info my-eval-run -o ./eval-outputs/

-o / --output now points at the run output root, not a metrics filename. Legacy files under .osmosis/metrics/ are left in place — delete them manually once you have re-run eval info under the new layout.

eval download

Download evaluation run outputs — metrics, trajectories, artifacts, and logs — from the platform to your local disk.

osmosis eval download <run-name> [--type TYPES] [--rows ROWS] [--output PATH] [--overwrite] [--yes]

Argument / Option	Type	Default	Description
`name_or_id`	`str` (required)	—	Evaluation run name or ID
`--type`	`str`	`metrics,trajectories`	Comma-separated selector: `metrics`, `trajectories`, `artifacts`, `logs`, or `all`. Replaces the default selection.
`--rows`	`str`	—	Rows to include for trajectories and artifacts, for example `3,7,10-20`. Each selected row includes all of its runs.
`-o`, `--output`	`path`	`.osmosis/evals/<name>/`	Run output root. Filenames and subdirectories below the root are fixed.
`--overwrite`	flag	—	Re-download files that already exist locally with a matching size.
`-y`, `--yes`	flag	—	Skip the confirmation prompt for downloads larger than 100 MiB.

Downloads land in a fixed run-scoped layout so re-running the command resumes cleanly:

.osmosis/evals/<run-name>/
├── metrics.json
├── summary.jsonl
├── trajectories/row_<row>_run_<n>.json
├── artifacts/row_<row>_run_<n>/...
└── logs.txt

Behavior:

The CLI skips files whose local size matches the platform manifest unless you pass --overwrite, so re-running the command only fetches missing or partial files.
Downloads over 100 MiB in total require confirmation; pass --yes to skip the prompt in scripts.
The CLI requests presigned URLs in bounded batches and downloads up to eight files concurrently. Each file writes to a *.partial sibling, then moves into place atomically once the CLI verifies its size.
The CLI retries failed files with backoff. If some files still fail, it reports each failed path and exits partial — re-run the command to retry only what is missing.
Pending runs fail early with a clear message; the platform performs authoritative path validation.

# Default: metrics + trajectories for the whole run
osmosis eval download my-eval-run

# Only artifacts for rows 3, 7, and 10–20
osmosis eval download my-eval-run --type artifacts --rows 3,7,10-20

# Everything, into a custom root, non-interactive
osmosis eval download my-eval-run --type all -o ./eval-outputs/ --yes

eval retry

Re-run a managed evaluation run’s failed and skipped samples. Graded samples are kept: the run resumes against its own sample index under the same run ID, so only the ungraded rows execute again.

osmosis eval retry <run-name> [--yes] [--secrets-file PATH]

Argument / Option	Type	Description
`name_or_id`	`str` (required)	Evaluation run name or ID
`-y`, `--yes`	flag	Skip the confirmation prompt
`--secrets-file`	`path`	Dotenv file supplying values the original run provided itself; `-` reads stdin. Values are never saved

The run must be finished, failed, or stopped, and must have at least one ungraded sample. A graded sample is carried forward and never re-run, even when its reward was below the pass threshold. A retry re-runs against the same model, dataset, and commit the run was submitted with. It bills like a new submission, metered as its own attempt. If the original run supplied secret values itself, Osmosis never stored those values. The platform answers with the names it needs, and the CLI resolves them from --secrets-file, the environment, or an interactive prompt before retrying. The values are sent with this request only and are redacted from any validation error the platform echoes back. Local runs retry on the machine that produced them. For a local run, this command prints its eval run --retry-failed --upload invocation instead of retrying, using the config path recorded in the run’s provenance. See eval run --retry-failed.

osmosis eval retry my-eval-run --yes
osmosis eval retry my-eval-run --secrets-file .env.run

eval stop

Stop a pending or running evaluation run.

osmosis eval stop <run-name> [--yes]

Argument / Option	Type	Description
`name_or_id`	`str` (required)	Evaluation run name or ID
`-y`, `--yes`	flag	Skip the confirmation prompt

eval logs

Show the most recent lifecycle logs for an evaluation run, oldest first. Use this to diagnose failed runs.

osmosis eval logs <run-name> [--limit N] [--cursor CURSOR]

Argument / Option	Type	Default	Description
`name_or_id`	`str` (required)	—	Evaluation run name or ID
`--limit`	`int`	`50`	Entries per page (1–200)
`--cursor`	`str`	—	Cursor from a previous page’s `next_cursor`

When older entries exist, --json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.

eval rubric

Run LLM-as-judge evaluation locally over a JSONL conversation file. This sub-command does not require a workspace directory or platform authentication and does not run a rollout.

osmosis eval rubric -d <data.jsonl> -r <rubric> --model <model>

Option	Type	Default	Description
`-d`, `--data`	`str` (required)	—	Path to JSONL conversations
`-r`, `--rubric`	`str` (required)	—	Rubric text or `@file.txt`
`--model`	`str` (required)	—	Judge model in LiteLLM format
`-n`, `--number`	`int`	`1`	Number of evaluation runs per record
`-o`, `--output`	`str`	—	Path for JSON output
`--api-key`	`str`	—	Deprecated; use the provider’s environment variable. The flag still works but emits a `DEPRECATION` warning.
`--timeout`	`float`	—	Request timeout in seconds
`--score-min`	`float`	`0.0`	Minimum score
`--score-max`	`float`	`1.0`	Maximum score

benchmark

Discover managed benchmarks and submit, inspect, and manage benchmark runs for the selected workspace. The complete command family works without a local repository when root --workspace is set. benchmark list and benchmark info act on benchmarks themselves: the workspace list and one benchmark’s page. Commands for an individual run live under benchmark runs.

benchmark list

List the benchmarks added to the current workspace.

osmosis benchmark list [--limit N] [--all]

Option	Type	Default	Description
`--limit`	`int`	`50`	Maximum number of benchmarks to show
`--all`	flag	off	Fetch every benchmark

The table shows Name, the shell-safe Key, Last Run, and Tasks. Last Run reads as the newest run’s state, its age, and its name (Finished · 2d ago · brave-otter). The key, the exact case-sensitive name, and the ID all work in [experiment].benchmark. JSON list items include run_count, running_count, last_run_at, last_run_status, last_run_name, and creator_name. A Harbor registry benchmark’s task list pages in from the registry after the benchmark is added. Until it finishes, Last Run reports the sync instead of a run (Syncing · 412 / 500 tasks), Tasks stays empty, and a run cannot be submitted against it. A benchmark whose sync failed reports the reason in Last Run, shows unavailable under Tasks, and returns a sync_error; retry its sync from the benchmark’s page in the Platform. The reported platform_url opens that page, with its leaderboard and runs.

benchmark info

Show a benchmark’s summary, leaderboard, and runs.

osmosis benchmark info <key_name_or_id> [--limit N] [--all]

Argument / Option	Type	Default	Description
`key_name_or_id`	`str` (required)	—	Benchmark key, exact name, or ID
`--limit`	`int`	`50`	Maximum number of runs to show in the runs table
`--all`	flag	off	Show every run in the runs table

The rich output opens with the benchmark summary (the source’s upstream page, runner, task and category counts, named task sets, harness and judge requirements, and pass threshold), followed by the benchmark’s leaderboard and a table of the workspace’s runs on it. --limit and --all apply to the runs table. When a benchmark’s published scores were measured on a specific harness, that harness is reported as the default. The LLM Judge row reads Required (default: <model>) when the run chooses the judge model, API key only (pinned grader) when the adapter pins its own grader and the run supplies only judge_api_key_secret, and – when the benchmark has no judge. JSON output also includes the complete task manifest and a leaderboard array:

osmosis --json benchmark info terminal-bench-2-1

Each leaderboard entry carries the rank and a tied flag (set when the paired test can’t separate that entrant from the top-ranked one, so it shares rank 1). It also reports the task set (full or parity), the harness and model, and the scoring run’s id, name, and platform_url. The five ranked metrics follow: pass_at_1 with its confidence interval, pass_at_k, the per-task cost_per_task, mean_duration_seconds, and tokens_per_task. Those are the same metrics the Platform’s leaderboard ranks by. Each task in the JSON manifest has a difficulty field with the value easy, medium, hard, or null. null means the source did not provide a difficulty, and clients must not infer one. For HLE, benchmark info also marks parity as the recommended named task set. Omit [tasks] from the config to select the full benchmark; use task_names, categories, or a listed task_set for a subset.

benchmark submit

Submit a benchmark run from any readable Osmosis TOML config path. Keeping configs under configs/benchmark/ is a repository convention, not a CLI requirement.

osmosis benchmark submit <config_path> [--yes] [--secrets-file PATH]

Argument / Option	Type	Description
`config_path`	`path` (required)	Any readable benchmark TOML file
`-y`, `--yes`	flag	Skip the confirmation prompt
`--secrets-file`	`path`	Dotenv file supplying values for `[secrets].required` names; `-` reads stdin

osmosis benchmark submit configs/benchmark/terminal-bench-smoke.toml

# Supply a per-run token without saving it to the secret store
osmosis benchmark submit configs/benchmark/hle-parity.toml --secrets-file .env.run

The benchmark named in [experiment].benchmark must already be added to the selected workspace. The command previews the selected tasks, agents, attempts, concurrency, and resolved secret scopes before submission. [secrets].required names use the same file → process environment → stored personal/workspace secret → hidden TTY prompt order as eval and training. Locally supplied values are labeled Run and sent with this submission’s TLS request, but are not added to the Osmosis secret store or persistent run config; stored values are resolved server-side. Every other secret reference (model api_key_secret, harness_api_key_secret, judge_api_key_secret, [verifier].required) must already exist in the workspace or personal scope. The result includes the generated run name, task count, status, platform_url, and selected workspace context. See Benchmarks for the CLI workflow and Configuration Files for the full TOML schema.

benchmark runs list

List benchmark runs for the current workspace. The table matches the Platform’s runs table: Name, Status, Progress, Benchmark, Agents, Best Pass@1, Submitted, and Submitted By.

osmosis benchmark runs list [--limit N] [--all]

Option	Type	Default	Description
`--limit`	`int`	`50`	Maximum number of benchmark runs to show
`--all`	flag	—	Show all benchmark runs

benchmark runs info

Show a benchmark run’s configuration, agents, progress, result totals, and metrics.

osmosis benchmark runs info <run-name>

Argument	Type	Description
`name_or_id`	`str` (required)	Benchmark run name or ID

The summary reports status, progress, duration, best pass@1, and submission details. The Agents section scores each agent the way the run’s Agent Results table does: rank, pass@1 with its interval, the deepest pass@k, and per-task cost, time, and tokens. Results totals report outcome counts, input and output tokens, and LLM Cost, which is model spend on your own provider keys and is not billed by Osmosis. The result includes platform_url, which opens the run in the Platform.

benchmark runs logs

Show recent lifecycle logs for a benchmark run, oldest first. Use this to monitor progress and diagnose failures.

osmosis benchmark runs logs <run-name> [--limit N] [--cursor CURSOR]

Argument / Option	Type	Default	Description
`name_or_id`	`str` (required)	—	Benchmark run name or ID
`--limit`	`int`	`50`	Entries per page (1–200)
`--cursor`	`str`	—	Cursor from a previous page’s `next_cursor`

When older entries exist, --json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.

benchmark runs stop

Stop a pending, queued, or running benchmark run.

osmosis benchmark runs stop <run-name> [--yes]

Argument / Option	Type	Description
`name_or_id`	`str` (required)	Benchmark run name or ID
`-y`, `--yes`	flag	Skip the confirmation prompt

benchmark runs download

Download benchmark summary metrics, task-level results, result artifacts, or logs.

osmosis benchmark runs download <run-name> [--type TYPES] [--output PATH] [--overwrite] [--yes]

Argument / Option	Type	Default	Description
`name_or_id`	`str` (required)	—	Benchmark run name or ID
`--type`	`str`	`summary,results`	Comma-separated selector: `summary`, `results`, `artifacts`, `logs`, or `all`. Replaces the default selection.
`-o`, `--output`	`path`	`.osmosis/benchmarks/<name>/`	Run output root. Filenames and subdirectories below the root are fixed.
`--overwrite`	flag	—	Re-download files that already exist locally with a matching size.
`-y`, `--yes`	flag	—	Skip the confirmation prompt for large downloads.

Downloads use a fixed run-scoped layout:

.osmosis/benchmarks/<run-name>/
├── summary.csv
├── results.csv
├── logs.txt
└── artifacts/<result-id>/<path>

The CLI skips complete local files unless --overwrite is set, writes partial downloads atomically, and reports any files that still fail after retrying. Re-run the same command to fetch only missing or incomplete files. pending and queued runs do not have downloadable outputs. Downloads from a running run are snapshots; use --overwrite to refresh existing files.

secret

Manage Platform environment_secret records. Train and eval configs reference them through [secrets].required. Benchmark configs reference model (api_key_secret), harness (harness_api_key_secret), and judge (judge_api_key_secret) records, along with any dataset-verifier records listed in [verifier].required. Osmosis never returns secret values to the CLI — these commands show or accept names and metadata only. Values are read from a hidden interactive prompt or a named environment variable, never as a plaintext command-line argument. Each secret has a scope:

Workspace secrets are shared across the workspace. Creating, updating, or deleting them requires an admin or owner role.
Personal secrets are private to you. When a workspace and a personal secret share a name, your personal value wins at run time. Use personal secrets for overrides, such as swapping in your own provider API key without affecting teammates.

Secret names must match ^[A-Z][A-Z0-9_]*$ (SCREAMING_SNAKE_CASE).

secret list

List secrets visible to you in the current workspace (names and metadata only — never values).

osmosis secret list [--scope all|workspace|personal] [--limit N] [--all]

Option	Type	Default	Description
`--scope`	`str`	`all`	Filter by scope: `all`, `workspace`, or `personal`
`--limit`	`int`	`50`	Maximum number of secrets to show
`--all`	flag	—	Show all secrets

The output includes a Scope column labeled Workspace or Personal.

osmosis secret list
osmosis secret list --scope personal

secret set

Create or update (upsert) a secret. The CLI reads the value from the env var named by --env VARNAME; without that flag, you type the value at a hidden interactive prompt. In --json or --plain (non-interactive) modes, you must pass --env.

osmosis secret set <name> [--scope workspace|personal] [--env VARNAME]

Argument / Option	Type	Default	Description
`name`	`str` (required)	—	Secret name in `^[A-Z][A-Z0-9_]*$`
`--scope`	`str`	`personal`	`personal` (private to you) or `workspace` (shared with the workspace)
`--env`	`str`	—	Read the value from this environment variable

# Create or update your personal secret (default scope):
osmosis secret set OPENAI_API_KEY

# Read the value from an env var (recommended for scripts and CI):
OPENAI_API_KEY=sk-... osmosis secret set OPENAI_API_KEY --env OPENAI_API_KEY

# Workspace-shared secret (requires admin or owner role):
OPENAI_API_KEY=sk-... osmosis secret set OPENAI_API_KEY --scope workspace --env OPENAI_API_KEY

secret delete

Delete a secret within the given scope. The CLI prompts you to confirm unless you pass --yes.

osmosis secret delete <name> [--scope workspace|personal] [--yes]

Argument / Option	Type	Default	Description
`name`	`str` (required)	—	Secret name to delete
`--scope`	`str`	`personal`	Scope of the secret to delete
`-y`, `--yes`	flag	—	Skip the confirmation prompt

osmosis secret delete OPENAI_API_KEY --yes
osmosis secret delete OPENAI_API_KEY --scope workspace --yes

dataset

Manage platform datasets for the selected workspace. Platform-backed commands accept root --workspace without a local repository; dataset validate is local and needs neither workspace scope nor platform authentication.

Command	Description
`osmosis dataset upload <file> [--overwrite] [--yes]`	Upload a CSV, JSONL, or Parquet dataset
`osmosis dataset download <name> [-o PATH] [--overwrite]`	Download a dataset file
`osmosis dataset list [--limit N] [--all]`	List datasets
`osmosis dataset info <name>`	Show dataset details and processing status
`osmosis dataset preview <name> [--rows N]`	Preview uploaded dataset rows
`osmosis dataset validate <file>`	Validate a dataset locally
`osmosis dataset logs <name> [--limit N] [--cursor CURSOR]`	Show lifecycle logs for a dataset (oldest first)

osmosis dataset upload data/train.jsonl
osmosis dataset info train
osmosis dataset preview train --rows 5

Dataset names are derived from the file name without its extension. If a dataset with that name already exists, the upload fails — pass --overwrite to replace it (the old record is soft-deleted). Use osmosis dataset logs <name> to diagnose failed uploads. --limit accepts 1–200 entries (default: 50). When older entries exist, the --json output includes a non-null next_cursor; pass it back with --cursor to page further back in time.

osmosis dataset upload requires --yes (-y) in non-interactive modes (--json, --plain, or when stdin is piped). Running without it raises INTERACTIVE_REQUIRED, even with --overwrite. Add --yes to CI jobs and scripted uploads to skip the confirmation prompt.

train

Submit source-backed training runs and manage runs for the selected workspace. List, info, logs, and stop accept root --workspace without a local repository.

train submit

osmosis train submit <config_path> [--yes] [--secrets-file PATH]

Argument / Option	Type	Description
`config_path`	`path` (required)	Training TOML under `configs/training/`
`-y`, `--yes`	flag	Skip confirmation prompt
`--secrets-file`	`path`	Dotenv file supplying values for `[secrets].required` names; `-` reads stdin

With root --workspace, config_path must be an absolute path under the containing repository’s configs/training/ directory. The CLI verifies that repository is connected to the selected workspace before submission:

osmosis --workspace <workspace-name> train submit /absolute/path/to/repository/configs/training/<name>.toml

Config values come from the local TOML file. Training code comes from the synced workspace repository. [secrets].required names use the same file → process environment → stored personal/workspace secret → hidden TTY prompt resolution order documented under eval submit; locally supplied values are sent with the current TLS request but are not added to the Osmosis secret store or persistent run config.

train info

osmosis train info <run-name> [--output PATH]

Shows run details, checkpoints, and metrics. Available checkpoints are listed while a run is running as well as after it ends; pending and queued runs have no checkpoint listing. The summary panel includes progress (current_step / total_steps with percent complete) and the latest reward while the run is in flight. In rich mode, metrics are saved under .osmosis/metrics/ by default.

train logs

Show the most recent lifecycle logs for a training run, oldest first. Use this to diagnose failed or crashed runs.

osmosis train logs <run-name> [--limit N] [--cursor CURSOR]

Argument / Option	Type	Default	Description
`name`	`str` (required)	—	Training run name
`--limit`	`int`	`50`	Entries per page (1–200)
`--cursor`	`str`	—	Cursor from a previous page’s `next_cursor`

When older entries exist, --json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.

Other Train Commands

osmosis train list [--limit N] [--all]
osmosis train stop <run-name> [--yes]

train list shows status, current step / total steps, and the latest reward for each run.

model

Manage base (foundation) models and LoRA models produced by training runs. Deploying a LoRA model exposes it for inference.

model list

List base models and LoRA models for the current workspace as two independently paginated sections (base first, then LoRA). The base table shows Name, Created, and Created By. The LoRA table shows Name, Base Model, Training Run, Checkpoint Step, Training Reward, and Created. When deployment info is available, the LoRA table also shows Deployment Status and a deployment-quota summary below it (for example, 2 of 5 inference deployments used).

osmosis model list [--type all|base|lora] [--limit N] [--all]

Option	Type	Default	Description
`--type`	`str`	`all`	Filter to a single list: `all`, `base`, or `lora`
`--limit`	`int`	`50`	Maximum number of models to show per type
`--all`	flag	—	Show all models of each type

--limit and --all apply to each list independently, and each list carries its own pagination cursor (next_offset).

osmosis model list
osmosis model list --type lora
osmosis model list --type base --limit 50
osmosis --json model list

--json output keys each list separately, so the structure itself identifies which list is which:

{
  "base_models": { "items": [...], "next_offset": null },
  "lora_models": { "items": [...], "next_offset": null },
  "active_deployments": 2,
  "max_active_deployments": 5
}

When deployment info is available, the active_deployments / max_active_deployments quota keys appear for --type all and --type lora, but not for --type base.

model info

Show details for a single model. A LoRA model prints base model, training run, checkpoint step, training reward, Hugging Face export status, and deployment status when deployment info is available. A base model prints its Hugging Face path, parameters, context window, Hugging Face URL, and inference prices when the model is priced for inference.

osmosis model info <model>

<model> is a LoRA model name, a base model name, or a base model Hugging Face path (Qwen/Qwen3-4B). Paths always resolve as base models; names try LoRA first, then base. If neither matches, the CLI reports Model not found: <model>.

osmosis model info qwen3-run1-step-100
osmosis model info Qwen/Qwen3-4B
osmosis --json model info Qwen/Qwen3-4B

--json output uses a base_model key for base models and a lora_model key for LoRA models, so the shape identifies which type was returned.

model deploy

Deploy or reactivate a LoRA model by name.

osmosis model deploy <lora-model>

model undeploy

Transition a LoRA model’s deployment to inactive (idempotent). The LoRA model remains in the training run history.

osmosis model undeploy <lora-model>

The standalone osmosis deployment, osmosis deploy, and osmosis undeploy commands have been removed. Use osmosis model deploy <lora-model> and osmosis model undeploy <lora-model> instead.

upgrade

Self-upgrade the CLI to the latest version published on PyPI.

osmosis upgrade

The CLI auto-detects your install method (pip, pipx, or uv tool) and runs the appropriate upgrade command.

​Workspace Scope

​Environment Variables

​quickstart

​auth

​doctor

​template

​template list

​template apply

​rollout

​rollout init

​rollout list

​eval

​eval run

​eval upload

​eval submit

​eval list

​eval info

​eval download

​eval retry

​eval stop

​eval logs

​eval rubric

​benchmark

​benchmark list

​benchmark info

​benchmark submit

​benchmark runs list

​benchmark runs info

​benchmark runs logs

​benchmark runs stop

​benchmark runs download

​secret

​secret list

​secret set

​secret delete

​dataset

​train

​train submit

​train info

​train logs

​Other Train Commands

​model

​model list

​model info

​model deploy

​model undeploy

​upgrade

Workspace Scope

Environment Variables

quickstart

auth

doctor

template

template list

template apply

rollout

rollout init

rollout list

eval

eval run

eval upload

eval submit

eval list

eval info

eval download

eval retry

eval stop

eval logs

eval rubric

benchmark

benchmark list

benchmark info

benchmark submit

benchmark runs list

benchmark runs info

benchmark runs logs

benchmark runs stop

benchmark runs download

secret

secret list

secret set

secret delete

dataset

train

train submit

train info

train logs

Other Train Commands

model

model list

model info

model deploy

model undeploy

upgrade