configs/eval/. Run locally through the backend the rollout entrypoint constructs, LocalBackend or Harbor, for fast iteration, optionally publish the completed result, or use managed infrastructure with eval submit.
Training pre-flight still requires a managed full-size run created by
osmosis eval submit. An uploaded local result is visible on the Platform but does not replace that gate.A Harbor sandbox that cannot reach this machine’s loopback interface — Daytona and Docker outside macOS — needs a public URL for the local model bridge.
osmosis eval run detects that from the rollout server’s health report and starts a managed cloudflared quick tunnel automatically, so keep that binary on PATH or pass --advertise-url with --listener-port for a tunnel you run yourself.Quick Start
From inside your workspace directory:--dataset-file form does not require platform credentials when --upload is omitted. Selecting the platform dataset named in the config or publishing with --upload still requires authenticated Git workspace context.
Local output defaults to .osmosis/evals/<run-name>/. You can publish a completed local run later by name, or by an explicit directory:
Evaluation Config
See Config Files for the full field reference.configs/eval/my-rollout.toml
When
[evaluation].limit is omitted, managed eval submit evaluates a random 10% sample of the dataset (at least one row), while local eval run evaluates every row unless --rows selects an explicit subset. Set limit to evaluate the first N rows in either mode.How It Works
Local Execution and Upload
Install the local-run dependencies withpython -m pip install "osmosis-ai[eval]>=0.3.4,<0.4". Every local run requires a valid local workspace and canonical eval config. A run with --dataset-file PATH and no --upload does not initialize platform credentials; selecting the platform dataset or uploading does. Provider and workflow credentials resolve from the process environment or --secrets-file; --secrets-file values take precedence for this run only, and the original environment is restored when the run exits.
The CLI replaces non-empty configured [secrets] values in logs.txt with [REDACTED]. It also redacts values of at least 8 characters from a known set of provider and platform environment variables, including OPENAI_API_KEY and OSMOSIS_TOKEN. Shorter values from those ambient variables are left unchanged unless they are configured secrets.
osmosis eval run executes local workspace files through the backend the rollout entrypoint constructs, LocalBackend or Harbor. Add --upload to publish only after a complete terminal run under the same run lock. Failed and skipped samples are terminal and uploadable; pending and cancelled runs are not.
osmosis eval upload <run-name> publishes an already-completed compatible local run and has no confirmation prompt. A bare run name resolves under the workspace’s .osmosis/evals/, while an explicit relative or absolute directory is used as given. That directory must contain compatible manifest.json, index.jsonl, progress.json, and metrics.json files, and the command requires authenticated workspace context. Upload is idempotent and server-authoritative. Re-running the same command after an interruption resumes the same platform run and uploads only the files still missing. That covers an unfinished upload. Publishing a different set of results for a local run that already finished importing is a separate case that needs --replace. See Retrying Failed Samples.
The payload contains index.jsonl, progress.json, the combined logs.txt when present, and only referenced canonical trajectory*.json files and safe artifacts for selected rollout IDs. The uploaded logs.txt appears in the run’s Logs tab on the platform. Before hashing the log, eval upload re-applies redaction using known provider and platform key values of at least 8 characters from the current process environment. This does not detect arbitrary secrets or old credentials no longer present in that environment; review older logs before publishing. The CLI never uploads the local manifest.json bytes, events.jsonl, metrics.json, summary or projection copies, control files, per-trial logs, or superseded attempts. Manifest digest, schema versions, and allowlisted redacted Git provenance are metadata. The server strictly validates files, recomputes metrics, and never launches hosted or Temporal evaluation work for an upload.
Uploaded results appear in eval list, eval info, and Platform viewers with a Local badge. A dirty source tree produces a provenance warning.
Local output uses the same metrics.json, summary.jsonl, trajectories/, and artifacts/ shape as eval download, plus a durable journal, local manifest, secrets-redacted logs, and the canonical rollout_trials/<rollout-id>/ store under .osmosis/evals/<run-name>/.
Omitting --name creates an adjective-animal-number name. Pass that name back to resume only work without a durable terminal result. A named run is locked to its resolved model, dataset, selected rows, evaluation settings, entrypoint, and rollout source; changed inputs refuse resume. Use --fresh to archive the old run before restarting, or --retry-failed without a code change to retry failures while keeping successes; see Retrying Failed Samples. If a named run is already complete, an interactive invocation offers a new generated-name run; non-interactive modes leave it untouched and print next steps.
The rollout server runs through uv in an environment resolved from rollouts/<name>/pyproject.toml, so declare Harbor and other rollout-side dependencies there. The first run syncs that environment and later runs reuse it. Upgrade both the CLI and the rollout dependency to 0.3.3 or later in the 0.3.x series: local eval now uses long polling, and the earlier callback protocol is incompatible. A ROLLOUT_SDK_VERSION_MISMATCH warning reports different installed versions; it is not a compatibility guarantee. eval run supports LocalBackend and Harbor with Docker or Daytona. Daytona and Docker outside macOS reach the local model bridge through the tunnel described above.
To develop a rollout against a local SDK checkout, point that same file at it:
eval run for those mechanics and for Cloudflare’s quick-tunnel creation rate limit.
Resume is journal-backed: terminal results returned by long polling are written and fsync-ed before they count as completed local work. After interruption, work without a durable terminal record remains pending and can run again. SDK 0.3.3 records rollout protocol 0.4 in the resume inputs, so local runs from the earlier callback protocol cannot resume. Use a new name or --fresh, which moves the previous directory to .osmosis/evals/<run-name>.archive-<utc-timestamp>/ rather than deleting it. See the 0.3.3 migration guide.
Managed Execution
1
Resolve workspace and config
Normally the CLI reads the evaluation TOML and resolves the workspace from the current Git
origin. With root --workspace and an absolute config path, it instead locates the config’s containing Osmosis Git workspace and verifies the selected platform workspace is connected to that same repository. In both cases it validates canonical config paths and imports the configured rollout entrypoint so its backend can validate itself. That import is best effort: it is skipped with a warning when the local environment cannot satisfy the rollout’s declared dependencies, and the platform validates the entrypoint after installing them. It does not scan the module namespace to discover workflow or grader classes. See Files in a Rollout for the SDK entrypoint contract and the local-execution warning.2
Submit to the platform
The CLI submits the evaluation run request. The platform resolves the selected
branch or commit_sha once, clones that commit from the connected workspace repository, and prepares the evaluation environment.3
Validate the model
Before evaluating any rows, the platform runs a pre-flight check that confirms
[experiment].model_path is reachable with your configured credentials. If the model is unreachable — wrong name, missing or invalid API key, or provider rate limiting — the run fails early instead of consuming evaluation resources. Provide the model’s provider API key by registering it with osmosis secret set and listing it under [secrets].required (see Configuration Files).4
Run the rollout server-side
The platform starts your rollout and drives
AgentWorkflow.run(ctx) for each selected dataset row using [experiment].model_path as the evaluation policy. The row’s ground_truth is exposed to Grader.grade(ctx) as ctx.label, while optional row metadata is exposed as ctx.metadata; the grader runs when either a label or metadata is present and can use either or both.5
Aggregate results
The platform aggregates rewards, pass rates, and per-row results. Use
osmosis eval info <name> (or osmosis --json eval info <name>) to inspect them.Commands
See the Command Reference for the full flag list.
Retrying Failed Samples
Hostedeval retry and --replace for local uploads require CLI version 0.3.4 or later.
Each attempt classifies a sample as success, failed, or skipped. The meaning of skipped depends on origin: a managed run records it when the attempt timed out with no grade, while a local run records it when the grader succeeded and set remove_sample to exclude the row. A retry re-runs the skipped and failed rows and carries every success forward. It produces one set of results rather than two runs to reconcile, and only the re-run samples consume compute. A graded sample is never re-run, even when its reward was below the pass threshold.
Managed runs retry in place:
--secrets-file, the environment, or an interactive prompt, and the detail page asks for them in the Retry dialog.
Local runs retry on the machine that produced them, then republish:
osmosis eval retry on a local run prints the same command rather than retrying, because a local retry needs the workspace the run came from.
--retry-failed re-dispatches the journal’s failed and skipped work items while keeping successes, and --upload replaces that run’s published results in place. The platform run keeps its ID and URL; its manifest digest must still match, which is what proves both attempts evaluated the same resolved inputs. To publish a retried run in a later, separate command, use osmosis eval upload <run-name> --replace.
From Evaluation Run to Training Run
1
Run a local evaluation
Run
osmosis eval run configs/eval/my-rollout.toml for smoke testing and iteration. Add --upload only if the completed result should appear on the Platform.2
Iterate on rollout code
Iterate against local files, then push the settled revision to the workspace repository. Use
branch for a feature branch or commit_sha to pin the managed evaluation.3
Pass the managed evaluation gate
Run
osmosis eval submit configs/eval/my-rollout.toml for the full-size managed evaluation, then use osmosis eval list and osmosis eval info <name> to track progress and inspect results.4
Submit a training run
Once evaluation run results look healthy, run
osmosis train submit configs/training/my-rollout.toml. See Training.Download Run Outputs
Once a run has data, useosmosis eval download to pull metrics, trajectories, artifacts, and logs to your local disk instead of clicking through the web UI.
--overwrite, and downloads over 100 MiB require confirmation unless you pass --yes. The CLI retries failed files automatically and lists anything still missing so a second run picks it up. See the Command Reference for the full flag list.
osmosis eval info -o now points at the same run output root, and rich-mode metrics exports save to .osmosis/evals/<name>/metrics.json. Existing files under .osmosis/metrics/ are left untouched.Local Rubric Scoring
osmosis eval rubric is a local utility for scoring an existing JSONL conversation file with an LLM judge. It does not require a workspace directory or platform authentication, and it does not run a rollout. “Local” describes where the command runs. For each record, the CLI sends the rubric, the scored assistant message, and any original_input, ground_truth, or metadata fields on that record to the provider behind --model. Review that provider’s privacy and retention terms before scoring sensitive data.
Next Steps
Config Files
Full reference for evaluation and training configuration files.
Git Sync
Push and sync rollout code before managed evaluation runs or training runs.
Training
Submit and manage a training run after the managed full-size evaluation gate passes.