osmosis -h to see all available commands. Every sub-command supports -h / --help.
The global output flags --json and --plain are position-independent — they work before or after the command:
--json for automation and AI agents. Use --plain for low-noise shell output.
quickstart
Interactive setup: sign in, clone your workspace, and get your first agent prompt.
The wizard authenticates the CLI, resolves which workspace to use, waits for the workspace repository to be connected and clones it, reports whether billing is set up, then asks what you want to do. Training a model, evaluating a model on a dataset, and running a benchmark each print an agent prompt for that goal; exploring prints documentation links instead.
When you run
osmosis quickstart from inside a workspace clone, the wizard picks up that workspace from the git remote. From an empty or unrelated directory, it lists the workspaces you belong to and asks you to pick one. Pass --workspace <name> to skip the picker and target a specific workspace by name. This is useful when you belong to more than one workspace.
It is safe to re-run: every step checks the current state before changing anything, so an already-configured workspace gets an all-clear report and goes straight to asking what you want to do next. Re-running from an empty directory reuses the last clone path recorded for that workspace when it still exists. The wizard needs an interactive terminal and runs in rich output mode only, so it is not available with --json or --plain — see Onboarding for the manual steps it automates.
auth
Manage CLI authentication.osmosis auth whoami from inside a workspace repository, the CLI also resolves the linked Platform workspace. --json reports it as a workspace object (id, name, role); other modes show Workspace and Role rows. Outside a workspace repository, or when the repository is not linked to a Platform workspace, workspace is null and the rest of the output is unchanged. The command always verifies the stored credential or OSMOSIS_TOKEN against the Platform, so it needs network access and it fails with AUTH_REQUIRED when you are logged out or your session has expired.
doctor
Inspect and optionally repair the workspace directory scaffold.
When you are logged in and the workspace directory has a Platform-connected
origin remote, doctor also reports the linked workspace (Linked workspace: <name>, or the workspace resource field in --json). The lookup is best-effort — offline or logged out, the field is null and doctor still works.
template
Add starter rollouts from the Osmosis workspace template catalog.template list
template apply
rollout
Create local rollout scaffolds and list synced rollouts.rollout init
rollouts/<name>/main.pyrollouts/<name>/pyproject.tomlrollouts/<name>/README.mdconfigs/eval/<name>.tomlconfigs/training/<name>.toml
rollout list
eval
Submit and manage evaluation runs, and run LLM-as-judge rubric scoring locally.eval submit
Submit an evaluation run from a TOML config underconfigs/eval/. The platform clones the workspace repository identified by the origin remote and runs the rollout server-side, so push your commits before submitting.
Each name in
[secrets].required resolves in this order: --secrets-file, the CLI process environment, an existing personal or workspace secret, then a hidden prompt when stdin is a TTY. Existing stored values are resolved server-side. Values supplied locally by file, environment, or prompt are sent with this submission’s TLS request but are not added to the Osmosis secret store or persistent run config, so supply them again for later runs. Outside a TTY, names still unresolved after the stored-secret check are reported all at once so CI shows the full gap. See [env] and [secrets] for the config-side rules.Config values come from the local TOML file. Rollout code comes from the synced workspace repository. Use
[experiment].branch or [experiment].commit_sha to select the source; they are mutually exclusive.eval list
List evaluation runs for the current workspace repository, including each run’s status, average reward, and progress.eval info
Show details, results, and metrics for an evaluation run.
The detail fields show name, status, progress (rows completed and percent), duration, pass rate, tokens used, submission and completion times, and the dataset, model, and rollout the run used. Two sections follow:
- Configuration — entrypoint, the resolved config values (
limit,n,batch_size,pass_threshold, timeouts), branch, short commit SHA, resolved secret scopes, and[env]keys. - Results — graded, passed, failed, and skipped counts, pass threshold, reward stats (mean, std, min, median, max), and pass@k.
eval info reports aggregates only. Use eval logs for platform logs and eval download for row-level trajectories, artifacts, metrics, and logs.
For completed runs, eval info also fetches metrics (duration, pass rate, tokens used, reward statistics, and pass@k) from the platform and exports them as JSON:
- In rich mode, the CLI saves metrics to
.osmosis/evals/<name>/metrics.jsonby default when metrics are available. - In JSON or plain mode, pass
-o/--outputto set the run output root; the CLI writesmetrics.jsoninside that directory. - The CLI creates parent directories automatically.
- If metrics are not yet available (for example, the run is still pending), the CLI prints a notice instead of writing a file.
-o / --output now points at the run output root, not a metrics filename. Legacy files under .osmosis/metrics/ are left in place — delete them manually once you have re-run eval info under the new layout.eval download
Download evaluation run outputs — metrics, trajectories, artifacts, and logs — from the platform to your local disk.
Downloads land in a fixed run-scoped layout so re-running the command resumes cleanly:
- The CLI skips files whose local size matches the platform manifest unless you pass
--overwrite, so re-running the command only fetches missing or partial files. - Downloads over 100 MiB in total require confirmation; pass
--yesto skip the prompt in scripts. - The CLI requests presigned URLs in bounded batches and downloads up to eight files concurrently. Each file writes to a
*.partialsibling, then moves into place atomically once the CLI verifies its size. - The CLI retries failed files with backoff. If some files still fail, it reports each failed path and exits partial — re-run the command to retry only what is missing.
- Pending runs fail early with a clear message; the platform performs authoritative path validation.
eval stop
Stop a pending or running evaluation run.eval logs
Show the most recent lifecycle logs for an evaluation run, oldest first. Use this to diagnose failed runs.
When older entries exist,
--json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.
eval rubric
Run LLM-as-judge evaluation locally over a JSONL conversation file. This sub-command does not require a workspace directory or platform authentication and does not run a rollout.benchmark
Discover managed benchmarks and submit, inspect, and manage benchmark runs for the current workspace.benchmark list and benchmark info act on benchmarks themselves: the workspace list and one benchmark’s page. Commands for an individual run live under benchmark runs.
benchmark list
List the benchmarks added to the current workspace.
The table mirrors the Platform’s benchmarks page: Name, the shell-safe Key, Last Run, Tasks, and Added By. Last Run reads as the newest run’s state, its age, and its name (
Finished · 2d ago · brave-otter). The key, the exact case-sensitive name, and the ID all work in [experiment].benchmark. JSON list items include run_count, running_count, last_run_at, last_run_status, last_run_name, and creator_name.
A Harbor registry benchmark’s task list pages in from the registry after the benchmark is added. Until it finishes, Last Run reports the sync instead of a run (Syncing · 412 / 500 tasks), Tasks stays empty, and a run cannot be submitted against it. A benchmark whose sync failed reports the reason in Last Run, shows unavailable under Tasks, and returns a sync_error; retry its sync from the benchmark’s page in the Platform. The reported platform_url opens that page, with its leaderboard and runs.
benchmark info
Show a benchmark’s summary, leaderboard, and runs.
The rich output opens with the benchmark summary (the source’s upstream page, runner, task and category counts, named task sets, harness and judge requirements, and pass threshold), followed by the benchmark’s leaderboard and a table of the workspace’s runs on it.
--limit and --all apply to the runs table. When a benchmark’s published scores were measured on a specific harness, that harness is reported as the default. The LLM Judge row reads Required (default: <model>) when the run chooses the judge model, API key only (pinned grader) when the adapter pins its own grader and the run supplies only judge_api_key_secret, and – when the benchmark has no judge. JSON output also includes the complete task manifest and a leaderboard array:
leaderboard entry carries the rank and a tied flag (set when the paired test can’t separate that entrant from the top-ranked one, so it shares rank 1). It also reports the task set (full or parity), the harness and model, and the scoring run’s id, name, and platform_url. The five ranked metrics follow: pass_at_1 with its confidence interval, pass_at_k, the per-task cost_per_task, mean_duration_seconds, and tokens_per_task. Those are the same metrics the Platform’s leaderboard ranks by.
Each task in the JSON manifest has a difficulty field with the value easy, medium, hard, or null. null means the source did not provide a difficulty, and clients must not infer one.
For HLE, benchmark info also marks parity as the recommended named task set. Omit [tasks] from the config to select the full benchmark; use task_names, categories, or a listed task_set for a subset.
benchmark submit
Submit a benchmark run from an Osmosis TOML config underconfigs/benchmark/.
[experiment].benchmark must already be added to the current workspace. The command previews the selected tasks, agents, attempts, concurrency, and resolved secret scopes before submission. [secrets].required names use the same file → process environment → stored personal/workspace secret → hidden TTY prompt order as eval and training. Locally supplied values are labeled Run and sent with this submission’s TLS request, but are not added to the Osmosis secret store or persistent run config; stored values are resolved server-side. Every other secret reference (model api_key_secret, harness_api_key_secret, judge_api_key_secret, [verifier].required) must already exist in the workspace or personal scope. The result includes the generated run name, task count, status, and platform_url.
See Benchmarks for the CLI workflow and Configuration Files for the full TOML schema.
benchmark runs list
List benchmark runs for the current workspace. The table matches the Platform’s runs table: Name, Status, Progress, Benchmark, Agents, Best Pass@1, Submitted, and Submitted By.benchmark runs info
Show a benchmark run’s configuration, agents, progress, result totals, and metrics.
The summary reports status, progress, duration, best pass@1, and submission details. The Agents section scores each agent the way the run’s Agent Results table does: rank, pass@1 with its interval, the deepest pass@k, and per-task cost, time, and tokens. Results totals report outcome counts, input and output tokens, and LLM Cost, which is model spend on your own provider keys and is not billed by Osmosis. The result includes
platform_url, which opens the run in the Platform.
benchmark runs logs
Show recent lifecycle logs for a benchmark run, oldest first. Use this to monitor progress and diagnose failures.
When older entries exist,
--json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.
benchmark runs stop
Stop a pending, queued, or running benchmark run.benchmark runs download
Download benchmark summary metrics, task-level results, result artifacts, or logs.
Downloads use a fixed run-scoped layout:
--overwrite is set, writes partial downloads atomically, and reports any files that still fail after retrying. Re-run the same command to fetch only missing or incomplete files.
pending and queued runs do not have downloadable outputs. Downloads from a running run are snapshots; use --overwrite to refresh existing files.
secret
Manage Platformenvironment_secret records. Train and eval configs reference them through [secrets].required. Benchmark configs reference model (api_key_secret), harness (harness_api_key_secret), and judge (judge_api_key_secret) records, along with any dataset-verifier records listed in [verifier].required. Osmosis never returns secret values to the CLI — these commands show or accept names and metadata only. Values are read from a hidden interactive prompt or a named environment variable, never as a plaintext command-line argument.
Each secret has a scope:
- Workspace secrets are shared across the workspace. Creating, updating, or deleting them requires an admin or owner role.
- Personal secrets are private to you. When a workspace and a personal secret share a name, your personal value wins at run time. Use personal secrets for overrides, such as swapping in your own provider API key without affecting teammates.
^[A-Z][A-Z0-9_]*$ (SCREAMING_SNAKE_CASE).
secret list
List secrets visible to you in the current workspace (names and metadata only — never values).
The output includes a Scope column labeled Workspace or Personal.
secret set
Create or update (upsert) a secret. The CLI reads the value from the env var named by--env VARNAME; without that flag, you type the value at a hidden interactive prompt. In --json or --plain (non-interactive) modes, you must pass --env.
secret delete
Delete a secret within the given scope. The CLI prompts you to confirm unless you pass--yes.
dataset
Manage platform datasets for the current workspace repository.--overwrite to replace it (the old record is soft-deleted).
Use osmosis dataset logs <name> to diagnose failed uploads. --limit accepts 1–200 entries (default: 50). When older entries exist, the --json output includes a non-null next_cursor; pass it back with --cursor to page further back in time.
osmosis dataset upload requires --yes (-y) in non-interactive modes (--json, --plain, or when stdin is piped). Running without it raises INTERACTIVE_REQUIRED, even with --overwrite. Add --yes to CI jobs and scripted uploads to skip the confirmation prompt.train
Submit and manage training runs for the current workspace repository.train submit
Config values come from the local TOML file. Training code comes from the synced workspace repository.
[secrets].required names use the same file → process environment → stored personal/workspace secret → hidden TTY prompt resolution order documented under eval submit; locally supplied values are sent with the current TLS request but are not added to the Osmosis secret store or persistent run config.train info
current_step / total_steps with percent complete) and the latest reward while the run is in flight. In rich mode, metrics are saved under .osmosis/metrics/ by default.
train logs
Show the most recent lifecycle logs for a training run, oldest first. Use this to diagnose failed or crashed runs.
When older entries exist,
--json output includes a non-null next_cursor. Pass it back with --cursor to page further back in time.
Other train commands
train list shows status, current step / total steps, and the latest reward for each run.
model
Manage base (foundation) models and LoRA models produced by training runs. Deploying a LoRA model exposes it for inference.model list
List base models and LoRA models for the current workspace as two independently paginated sections (base first, then LoRA). The base table shows Name, Created, and Created By. The LoRA table shows Name, Base Model, Training Run, Checkpoint Step, Training Reward, and Created. When deployment info is available, the LoRA table also shows Deployment Status and a deployment-quota summary below it (for example,2 of 5 inference deployments used).
--limit and --all apply to each list independently, and each list carries its own pagination cursor (next_offset).
--json output keys each list separately, so the structure itself identifies which list is which:
active_deployments / max_active_deployments quota keys appear for --type all and --type lora, but not for --type base.
model info
Show details for a single LoRA model: base model, training run, checkpoint step, training reward, Hugging Face export status, and deployment status when deployment info is available.model deploy
Deploy or reactivate a LoRA model by name.model undeploy
Transition a LoRA model’s deployment to inactive (idempotent). The LoRA model remains in the training run history.The standalone
osmosis deployment, osmosis deploy, and osmosis undeploy commands have been removed. Use osmosis model deploy <lora-model> and osmosis model undeploy <lora-model> instead.upgrade
Self-upgrade the CLI to the latest version published on PyPI.pip, pipx, or uv tool) and runs the appropriate upgrade command.