Evaluation Runs

An evaluation run executes your rollout’s AgentWorkflow and Grader against a dataset, then reports aggregate scores and per-sample results. Runs visible on the Platform can originate from managed execution after eval submit or from a completed local eval run published with --upload or eval upload.

Concepts

Smoke Test or Formal Evaluation

Local smoke test: use osmosis eval run with a small [evaluation].limit to confirm that the rollout works end-to-end and the grader returns useful scores.
Managed formal evaluation before training: use osmosis eval submit and set [evaluation].limit to the dataset row count to evaluate every row; otherwise Osmosis selects a random 10% sample. An uploaded local result does not replace this managed full-size gate.

Evaluation Configuration vs Evaluation Run

An Evaluation Configuration identifies the model, dataset, AgentWorkflow, and evaluation settings. An Evaluation Run is one execution of that configuration. Submit the same configuration more than once to compare changes over time.

Dataset Input Modes

In prompt mode, dataset rows provide user_prompt and a reference answer in ground_truth or its label alias. In metadata mode, each row provides a metadata JSON object so the AgentWorkflow can construct the task. A Grader in metadata mode can use ctx.label, ctx.metadata, or both; write the workflow and grader to the contract of the dataset you select.

Publishing a Local Run

Run locally through the rollout’s LocalBackend or Harbor backend and publish after completion:

osmosis eval run configs/eval/my-rollout.toml --upload

You can publish a completed local run later by name, or by an explicit directory:

osmosis eval upload <run-name>
osmosis eval upload .osmosis/evals/<run-name>/

eval upload has no confirmation prompt. It requires authenticated workspace context and a compatible completed directory containing manifest.json, index.jsonl, progress.json, and metrics.json. Pending and cancelled runs cannot be uploaded; failed and skipped samples are terminal and uploadable. Publication is idempotent and server-authoritative. Re-running the same command after interruption returns the same platform run and uploads only missing server files. Resuming an interrupted upload needs nothing extra. Publishing a different set of results for a run that already finished importing needs --replace, which is how a retried run replaces what it published. When present, the combined logs.txt is uploaded to the run’s Logs tab. Before hashing it, the CLI re-applies redaction using known provider and platform key values of at least 8 characters from the current process environment; see log redaction and its limits. Osmosis strictly validates the allowed trajectories and artifacts and recomputes metrics without launching hosted or Temporal evaluation work.

Submitting a Managed Evaluation Run

Submit a TOML configuration from configs/eval/:

osmosis eval submit configs/eval/my-rollout.toml

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

Pass --yes to skip the confirmation prompt in scripts or CI:

osmosis eval submit configs/eval/my-rollout.toml --yes

Key Configuration Fields

[experiment]
rollout = "my-rollout"                  # Rollout directory name (under rollouts/)
entrypoint = "main.py"                  # Entrypoint file name
model_path = "openai/gpt-5-mini"        # LiteLLM-style model name
dataset = "my-dataset"                  # Platform dataset name
# branch = "my-feature"                 # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..."              # Optional: pin to a specific synced commit

[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200                           # First N rows; omit for a random 10% sample
# n = 1                                 # Evaluation attempts per row
# batch_size = 1                        # Rows evaluated per batch
# pass_threshold = 1.0                  # Minimum passing score
# agent_workflow_timeout_s = 450        # Agent workflow timeout per row
# grader_timeout_s = 150                # Grader timeout per row

branch and commit_sha are mutually exclusive. With branch, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository’s default branch.

See Config Files for the full TOML reference, including [env] and [secrets].

n is the number of attempts per selected dataset row. With limit = L and n = N, Osmosis runs up to L * N evaluations, or sampled_rows * n when sampling.

Status Lifecycle

Status	Description
pending	The run was submitted and is waiting to start.
running	The evaluation is executing against the selected rows.
finished	The evaluation completed successfully, and aggregate and sample results are available.
failed	The run ended because of an execution error. Check sample results and logs.
stopped	Someone stopped the run before it completed.

Monitoring

Platform Dashboard

Open the standalone Evaluation Runs page at platform.osmosis.ai to search and filter runs by status, dataset, model, and rollout. Uploaded local results carry a Local badge, and dirty Git provenance produces a warning. Each run page has a sidebar for status, sample progress, duration, pass rate, tokens used, submission details, dataset, model, and rollout, when those values are available. The run page has four tabs:

Overview: aggregate reward statistics and a pass@k view when the run has enough attempts.
Evaluation Samples: searchable, filterable sample outcomes with rewards, workflow messages, ground truth, grader metrics, and downloadable artifacts when present.
Configuration: the rollout entrypoint, pinned branch or commit, model, dataset, evaluation settings, resolved secret scopes, and environment keys.
Logs: lifecycle events and diagnostics for the run.

CLI Commands

# List evaluation runs for the current workspace repository
osmosis eval list
osmosis eval list --all

# Show details and the current result snapshot for one run
osmosis eval info my-eval-run

eval info reports run context, timing, progress, configuration, and aggregate results as they become available.

Managing Runs

Stopping a Run

osmosis eval stop my-eval-run

This requests a stop for a pending or running evaluation. The run moves to stopped after cleanup finishes. Pass --yes to skip the confirmation prompt.

Retrying Failed Samples

Osmosis classifies each sample attempt as success, failed, or skipped. The meaning of skipped depends on origin. A managed run records it when the attempt timed out with no grade. A local run records it when the grader succeeded and set remove_sample to exclude the row. A retry re-runs the skipped and failed rows and carries every success forward. This keeps one run rather than two to reconcile, and only the re-run samples consume compute. Retry run on a finished, failed, or stopped run’s detail page re-runs its ungraded samples. The same thing from the CLI:

osmosis eval retry my-eval-run

A managed run retries in place: it keeps its ID, URL, and configuration, and re-executes against the same model, dataset, and commit it was submitted with. A retry bills like a new submission, metered as its own attempt. A graded sample is carried forward and never re-run, even when its reward was below the pass threshold, so a retry can only complete a measurement rather than improve one. If the original run supplied secret values itself, those were never stored, so the Retry dialog asks for them again with the names filled in. A run that took several attempts carries a Retried badge showing how many samples were carried forward. A local run retries on the machine that produced it. Retry run on its detail page hands you the command:

osmosis eval run configs/eval/my-rollout.toml --name my-eval-run --retry-failed --upload

The page fills in the run’s own config path, recorded in its provenance at upload. --retry-failed --upload together replace the run’s published results in place, so the platform run keeps its ID and URL. See Retrying Failed Samples for the local details.

Next Steps

Config Files

Reference for the evaluation TOML config.

Datasets

Upload and validate datasets for evaluation runs.

Training Runs

Submit a training run after the managed full-size evaluation gate passes.

Get started

Runs

Resources

Workspace

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Publishing a Local Run

Submitting a Managed Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Retrying Failed Samples

Next Steps

Config Files

Datasets

Training Runs

​Concepts

​Smoke Test or Formal Evaluation

​Evaluation Configuration vs Evaluation Run

​Dataset Input Modes

​Publishing a Local Run

​Submitting a Managed Evaluation Run

​Key Configuration Fields

​Status Lifecycle

​Monitoring

​Platform Dashboard

​CLI Commands

​Managing Runs

​Stopping a Run

​Retrying Failed Samples

​Next Steps

Config Files

Datasets

Training Runs

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Publishing a Local Run

Submitting a Managed Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Retrying Failed Samples

Next Steps