Osmosis CLI Evaluation

Use an evaluation run to test a rollout and grader against a dataset. Evaluation configs live under configs/eval/. Run locally through the backend the rollout entrypoint constructs, LocalBackend or Harbor, for fast iteration, optionally publish the completed result, or use managed infrastructure with eval submit.

Training pre-flight still requires a managed full-size run created by osmosis eval submit. An uploaded local result is visible on the Platform but does not replace that gate.

A Harbor sandbox that cannot reach this machine’s loopback interface — Daytona, SkyPilot, and Docker outside macOS — needs a public URL for the local model bridge. osmosis eval run detects that from the rollout server’s health report and starts a managed cloudflared quick tunnel automatically, so keep that binary on PATH or pass --advertise-url with --listener-port for a tunnel you run yourself.

Quick Start

From inside your workspace directory:

osmosis eval run configs/eval/my-rollout.toml
osmosis eval run configs/eval/my-rollout.toml --dataset-file data/my-eval.jsonl
osmosis eval run configs/eval/my-rollout.toml --upload

The --dataset-file form does not require platform credentials when --upload is omitted. Selecting the platform dataset named in the config or publishing with --upload still requires authenticated Git workspace context. Local output defaults to .osmosis/evals/<run-name>/. You can publish a completed local run later by name, or by an explicit directory:

osmosis eval upload <run-name>
osmosis eval upload .osmosis/evals/<run-name>/

For a managed run, push the intended revision and submit:

git push
osmosis eval submit configs/eval/my-rollout.toml

From another current directory, select the workspace explicitly and pass the absolute canonical config path:

osmosis --workspace <workspace-name> eval submit /absolute/path/to/repository/configs/eval/my-rollout.toml

Then inspect or manage platform-visible runs:

osmosis eval list
osmosis eval info <name>
osmosis eval stop <name>

Evaluation Config

See Config Files for the full field reference.

configs/eval/my-rollout.toml

[experiment]
rollout = "my-rollout"                # Rollout directory under rollouts/
entrypoint = "main.py"                # Entrypoint relative to the rollout directory
model_path = "openai/gpt-5-mini"      # LiteLLM-style model name for the evaluation policy
dataset = "my-platform-dataset"       # Platform dataset name from `osmosis dataset list`
# branch = "my-feature"               # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc1234"             # Optional: pin to a specific commit

[evaluation]
# Optional. Omit values to use defaults for the selected execution mode.
# limit = 200
# n = 1
# batch_size = 1
# pass_threshold = 1.0
# agent_workflow_timeout_s = 450
# grader_timeout_s = 150

# [env]
# LOG_LEVEL = "INFO"

[secrets]
# Required for eval configs. Use required = [] only when no secrets are needed.
required = ["OPENAI_API_KEY"]

When [evaluation].limit is omitted, managed eval submit evaluates a random 10% sample of the dataset (at least one row), while local eval run evaluates every row unless --rows selects an explicit subset. Set limit to evaluate the first N rows in either mode.

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

How It Works

Local Execution and Upload

Install the local-run dependencies with python -m pip install "osmosis-ai[eval]>=0.3.2,<0.4". Every local run requires a valid local workspace and canonical eval config. A run with --dataset-file PATH and no --upload does not initialize platform credentials; selecting the platform dataset or uploading does. Provider and workflow credentials resolve from the process environment or --secrets-file and stay on your machine; --secrets-file values take precedence for this run only, and the original environment is restored when the run exits. osmosis eval run executes local workspace files through the backend the rollout entrypoint constructs, LocalBackend or Harbor. Add --upload to publish only after a complete terminal run under the same run lock. Failed and skipped samples are terminal and uploadable; pending and cancelled runs are not. osmosis eval upload <run-name> publishes an already-completed compatible local run and takes no confirmation or extra flags. A bare run name resolves under the workspace’s .osmosis/evals/, while an explicit relative or absolute directory is used as given. That directory must contain compatible manifest.json, index.jsonl, progress.json, and metrics.json files, and the command requires authenticated workspace context. Upload is idempotent and server-authoritative: after interruption, run the same command again to return the same platform run and upload only files missing on the server. The payload contains index.jsonl, progress.json, and only referenced canonical trajectory*.json files and safe artifacts for selected rollout IDs. The CLI keeps logs.txt local and never uploads the local manifest.json bytes, events.jsonl, metrics.json, summary or projection copies, control files, per-trial logs, or superseded attempts. Manifest digest, schema versions, and allowlisted redacted Git provenance are metadata. The server strictly validates files, recomputes metrics, and never launches hosted or Temporal evaluation work for an upload. Uploaded results appear in eval list, eval info, and Platform viewers with a Local badge. A dirty source tree produces a provenance warning. Local output uses the same metrics.json, summary.jsonl, trajectories/, and artifacts/ shape as eval download, plus a durable journal, local manifest, secrets-redacted logs, and the canonical rollout_trials/<rollout-id>/ store under .osmosis/evals/<run-name>/. Omitting --name creates an adjective-animal-number name. Pass that name back to resume only work without a durable terminal result. A named run is locked to its resolved model, dataset, selected rows, evaluation settings, entrypoint, and rollout source; changed inputs refuse resume. Use --fresh to archive the old run before restarting, or --retry-failed without a code change to retry failures while keeping successes. If a named run is already complete, an interactive invocation offers a new generated-name run; non-interactive modes leave it untouched and print next steps. The rollout server runs through uv in an environment resolved from rollouts/<name>/pyproject.toml, so declare Harbor and other rollout-side dependencies there. The first run syncs that environment and later runs reuse it, and the CLI prints a non-fatal ROLLOUT_SDK_VERSION_MISMATCH warning when that environment’s osmosis-ai version differs from its own. eval run supports LocalBackend and every Harbor environment; Daytona, SkyPilot, and Docker outside macOS reach the local model bridge through the tunnel described above, which an environment pinned below 0.3.2 cannot request on its own — pass --tunnel cloudflared there. To develop a rollout against a local SDK checkout, point that same file at it:

[tool.uv.sources]
osmosis-ai = { path = "/path/to/osmosis-sdk-python", editable = true }

Over a tunnel, only the sandbox’s OpenAI-compatible chat traffic leaves your machine: callbacks stay on loopback, every listener route requires a bearer, and the controller and model-bridge tokens are regenerated for each run. Tunnel mode also keeps a slow non-streaming model call alive past the proxy’s idle-read timeout. See eval run for those mechanics and for Cloudflare’s quick-tunnel creation rate limit. Resume is journal-backed: each terminal result is written and fsync-ed before the rollout server’s callback is acknowledged, so an interrupted run never loses an acknowledged result and never silently skips an unacknowledged one. --fresh moves the previous directory to .osmosis/evals/<run-name>.archive-<utc-timestamp>/ rather than deleting it.

Managed Execution

Resolve workspace and config

Normally the CLI reads the evaluation TOML and resolves the workspace from the current Git origin. With root --workspace and an absolute config path, it instead locates the config’s containing Osmosis Git workspace and verifies the selected platform workspace is connected to that same repository. In both cases it validates canonical config paths and imports the configured rollout entrypoint so its backend can validate itself. That import is best effort: it is skipped with a warning when the local environment cannot satisfy the rollout’s declared dependencies, and the platform validates the entrypoint after installing them. It does not scan the module namespace to discover workflow or grader classes. See Files in a Rollout for the SDK entrypoint contract and the local-execution warning.

Submit to the platform

The CLI submits the evaluation run request. The platform resolves the selected branch or commit_sha once, clones that commit from the connected workspace repository, and prepares the evaluation environment.

Validate the model

Before evaluating any rows, the platform runs a pre-flight check that confirms [experiment].model_path is reachable with your configured credentials. If the model is unreachable — wrong name, missing or invalid API key, or provider rate limiting — the run fails early instead of consuming evaluation resources. Provide the model’s provider API key by registering it with osmosis secret set and listing it under [secrets].required (see Configuration Files).

Run the rollout server-side

The platform starts your rollout and drives AgentWorkflow.run(ctx) for each selected dataset row using [experiment].model_path as the evaluation policy. The row’s ground_truth is exposed to Grader.grade(ctx) as ctx.label, while optional row metadata is exposed as ctx.metadata; the grader runs when either a label or metadata is present and can use either or both.

Aggregate results

The platform aggregates rewards, pass rates, and per-row results. Use osmosis eval info <name> (or osmosis --json eval info <name>) to inspect them.

Commands

Command	Description
`osmosis eval run <config>.toml [--upload]`	Execute an evaluation locally and optionally publish it after completion; see the command reference for existing run options.
`osmosis eval upload <run-name>`	Publish an already-completed compatible local run by name or run directory.
`osmosis eval submit <config>.toml [--yes]`	Submit an evaluation run from a TOML under the source repository’s `configs/eval/`.
`osmosis eval list [--limit N] [--all]`	List evaluation runs for the selected workspace.
`osmosis eval info <run-name>`	Show details and results for a specific evaluation run.
`osmosis eval download <run-name>`	Download metrics, trajectories, artifacts, and logs for a run.
`osmosis eval stop <run-name> [--yes]`	Stop a pending or running evaluation run.
`osmosis eval rubric`	Local LLM-as-judge over a JSONL conversation file. Does not touch the Osmosis platform.

See the Command Reference for the full flag list.

From Evaluation Run to Training Run

Run a local evaluation

Run osmosis eval run configs/eval/my-rollout.toml for smoke testing and iteration. Add --upload only if the completed result should appear on the Platform.

Iterate on rollout code

Iterate against local files, then push the settled revision to the workspace repository. Use branch for a feature branch or commit_sha to pin the managed evaluation.

Pass the managed evaluation gate

Run osmosis eval submit configs/eval/my-rollout.toml for the full-size managed evaluation, then use osmosis eval list and osmosis eval info <name> to track progress and inspect results.

Submit a training run

Once evaluation run results look healthy, run osmosis train submit configs/training/my-rollout.toml. See Training.

Download Run Outputs

Once a run has data, use osmosis eval download to pull metrics, trajectories, artifacts, and logs to your local disk instead of clicking through the web UI.

# Default: metrics + trajectories for the whole run
osmosis eval download my-eval-run

# Only trajectories and artifacts for selected rows
osmosis eval download my-eval-run --type trajectories,artifacts --rows 3,7,10-20

# Everything, into a custom root
osmosis eval download my-eval-run --type all -o ./eval-outputs/

Files land under a fixed layout so re-running the command resumes cleanly:

.osmosis/evals/<run-name>/
├── metrics.json
├── summary.jsonl
├── trajectories/row_<row>_run_<n>.json
├── artifacts/row_<row>_run_<n>/...
└── logs.txt

The CLI skips local files whose size matches the platform manifest unless you pass --overwrite, and downloads over 100 MiB require confirmation unless you pass --yes. The CLI retries failed files automatically and lists anything still missing so a second run picks it up. See the Command Reference for the full flag list.

osmosis eval info -o now points at the same run output root, and rich-mode metrics exports save to .osmosis/evals/<name>/metrics.json. Existing files under .osmosis/metrics/ are left untouched.

Local Rubric Scoring

osmosis eval rubric is a local utility for scoring an existing JSONL conversation file with an LLM judge. It does not require a workspace directory or platform authentication, and it does not run a rollout. “Local” describes where the command runs: for each record the CLI sends the rubric and the scored assistant message — plus any original_input, ground_truth, or metadata fields on that record — to the provider behind --model, so review that provider’s privacy and retention terms before scoring sensitive data.

osmosis eval rubric -d conversations.jsonl \
  --rubric "Evaluate the assistant's helpfulness..." \
  --model openai/gpt-5-mini

See the Command Reference for the full flag list.

Next Steps

Config Files

Full reference for evaluation and training configuration files.

Git Sync

Push and sync rollout code before managed evaluation runs or training runs.

Training

Submit and manage a training run after the managed full-size evaluation gate passes.

Get started

Runs

Workspace

Reference

Quick Start

Evaluation Config

How It Works

Local Execution and Upload

Managed Execution

Commands

From Evaluation Run to Training Run

Download Run Outputs

Local Rubric Scoring

Next Steps

Config Files

Git Sync

Training

​Quick Start

​Evaluation Config

​How It Works

​Local Execution and Upload

​Managed Execution

​Commands

​From Evaluation Run to Training Run

​Download Run Outputs

​Local Rubric Scoring

​Next Steps

Config Files

Git Sync

Training

Quick Start

Evaluation Config

How It Works

Local Execution and Upload

Managed Execution

Commands

From Evaluation Run to Training Run

Download Run Outputs

Local Rubric Scoring

Next Steps