Evaluation Runs

An evaluation run executes your rollout’s AgentWorkflow and Grader against a dataset, then reports aggregate scores and per-sample results. Runs visible on the Platform can originate from managed execution after eval submit or from a completed local eval run published with --upload or eval upload.

Concepts

Smoke Test or Formal Evaluation

Local smoke test: use osmosis eval run with a small [evaluation].limit to confirm that the rollout works end-to-end and the grader returns useful scores.
Managed formal evaluation before training: use osmosis eval submit and set [evaluation].limit to the dataset row count to evaluate every row; otherwise Osmosis selects a random 10% sample. An uploaded local result does not replace this managed full-size gate.

Evaluation Configuration vs Evaluation Run

An Evaluation Configuration identifies the model, dataset, AgentWorkflow, and evaluation settings. An Evaluation Run is one execution of that configuration. Submit the same configuration more than once to compare changes over time.

Dataset Input Modes

In prompt mode, dataset rows provide user_prompt and a reference answer in ground_truth or its label alias. In metadata mode, each row provides a metadata JSON object so the AgentWorkflow can construct the task. A Grader in metadata mode can use ctx.label, ctx.metadata, or both; write the workflow and grader to the contract of the dataset you select.

Publishing a Local Run

Run locally through the rollout’s LocalBackend or Harbor Docker backend and publish after completion:

osmosis eval run configs/eval/my-rollout.toml --upload

You can publish the default local run directory later:

osmosis eval upload .osmosis/evals/<run-name>/

eval upload takes no confirmation or extra flags. It requires authenticated workspace context and a compatible completed directory containing manifest.json, index.jsonl, progress.json, and metrics.json. Pending and cancelled runs cannot be uploaded; failed and skipped samples are terminal and uploadable. Publication is idempotent and server-authoritative. Re-running the same command after interruption returns the same platform run and uploads only missing server files. The CLI keeps logs.txt local; Osmosis strictly validates the allowed trajectories and artifacts and recomputes metrics without launching hosted or Temporal evaluation work.

Submitting a Managed Evaluation Run

Submit a TOML configuration from configs/eval/:

osmosis eval submit configs/eval/my-rollout.toml

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

Pass --yes to skip the confirmation prompt in scripts or CI:

osmosis eval submit configs/eval/my-rollout.toml --yes

Key Configuration Fields

[experiment]
rollout = "my-rollout"                  # Rollout directory name (under rollouts/)
entrypoint = "main.py"                  # Entrypoint file name
model_path = "openai/gpt-5-mini"        # LiteLLM-style model name
dataset = "my-dataset"                  # Platform dataset name
# branch = "my-feature"                 # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..."              # Optional: pin to a specific synced commit

[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200                           # First N rows; omit for a random 10% sample
# n = 1                                 # Evaluation attempts per row
# batch_size = 1                        # Rows evaluated per batch
# pass_threshold = 1.0                  # Minimum passing score
# agent_workflow_timeout_s = 450        # Agent workflow timeout per row
# grader_timeout_s = 150                # Grader timeout per row

branch and commit_sha are mutually exclusive. With branch, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository’s default branch.

See Config Files for the full TOML reference, including [env] and [secrets].

n is the number of attempts per selected dataset row. With limit = L and n = N, Osmosis runs up to L * N evaluations, or sampled_rows * n when sampling.

Status Lifecycle

Status	Description
pending	The run was submitted and is waiting to start.
running	The evaluation is executing against the selected rows.
finished	The evaluation completed successfully, and aggregate and sample results are available.
failed	The run ended because of an execution error. Check sample results and logs.
stopped	Someone stopped the run before it completed.

Monitoring

Platform Dashboard

Open the standalone Evaluation Runs page at platform.osmosis.ai to search and filter runs by status, dataset, model, and rollout. Uploaded local results carry a Local badge, and dirty Git provenance produces a warning. Each run page has a sidebar for status, sample progress, duration, pass rate, tokens used, submission details, dataset, model, and rollout, when those values are available. The run page has four tabs:

Overview: aggregate reward statistics and a pass@k view when the run has enough attempts.
Evaluation Samples: searchable, filterable sample outcomes with rewards, workflow messages, ground truth, grader metrics, and downloadable artifacts when present.
Configuration: the rollout entrypoint, pinned branch or commit, model, dataset, evaluation settings, resolved secret scopes, and environment keys.
Logs: lifecycle events and diagnostics for the run.

CLI Commands

# List evaluation runs for the current workspace repository
osmosis eval list
osmosis eval list --all

# Show details and the current result snapshot for one run
osmosis eval info my-eval-run

eval info reports run context, timing, progress, configuration, and aggregate results as they become available.

Managing Runs

Stopping a Run

osmosis eval stop my-eval-run

This requests a stop for a pending or running evaluation. The run moves to stopped after cleanup finishes. Pass --yes to skip the confirmation prompt.

Next Steps

Config Files

Reference for the evaluation TOML config.

Datasets

Upload and validate datasets for evaluation runs.

Training Runs

Submit a training run after the managed full-size evaluation gate passes.

Get started

Runs

Resources

Workspace

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Publishing a Local Run

Submitting a Managed Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Next Steps

Config Files

Datasets

Training Runs

​Concepts

​Smoke Test or Formal Evaluation

​Evaluation Configuration vs Evaluation Run

​Dataset Input Modes

​Publishing a Local Run

​Submitting a Managed Evaluation Run

​Key Configuration Fields

​Status Lifecycle

​Monitoring

​Platform Dashboard

​CLI Commands

​Managing Runs

​Stopping a Run

​Next Steps

Config Files

Datasets

Training Runs

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Publishing a Local Run

Submitting a Managed Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Next Steps