Evaluation Runs

An evaluation run executes your rollout’s AgentWorkflow and Grader against a platform dataset, then reports aggregate scores and per-sample results. Osmosis pulls the rollout from your synced workspace repository and runs the evaluation on managed infrastructure; no training run is required.

Concepts

Smoke Test or Formal Evaluation

Smoke test before training: use a small [evaluation].limit to confirm that the rollout works end-to-end and the grader returns useful scores before submitting a training run.
Formal evaluation: compare models or prompts, track quality over time, or run checks from CI. Set [evaluation].limit to the dataset row count to evaluate every row; otherwise Osmosis selects a random 10% sample.

Evaluation Configuration vs Evaluation Run

An Evaluation Configuration identifies the model, dataset, AgentWorkflow, and evaluation settings. An Evaluation Run is one execution of that configuration. Submit the same configuration more than once to compare changes over time.

Dataset Input Modes

In prompt mode, dataset rows provide user_prompt and a reference answer in ground_truth or its label alias. In metadata mode, each row provides a metadata JSON object so the AgentWorkflow can construct the task. A Grader in metadata mode can use ctx.label, ctx.metadata, or both; write the workflow and grader to the contract of the dataset you select.

Submitting an Evaluation Run

Submit a TOML configuration from configs/eval/:

osmosis eval submit configs/eval/my-rollout.toml

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

Pass --yes to skip the confirmation prompt in scripts or CI:

osmosis eval submit configs/eval/my-rollout.toml --yes

Key Configuration Fields

[experiment]
rollout = "my-rollout"                  # Rollout directory name (under rollouts/)
entrypoint = "main.py"                  # Entrypoint file name
model_path = "openai/gpt-5-mini"        # LiteLLM-style model name
dataset = "my-dataset"                  # Platform dataset name
# branch = "my-feature"                 # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..."              # Optional: pin to a specific synced commit

[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200                           # First N rows; omit for a random 10% sample
# n = 1                                 # Evaluation attempts per row
# batch_size = 1                        # Rows evaluated per batch
# pass_threshold = 1.0                  # Minimum passing score
# agent_workflow_timeout_s = 450        # Agent workflow timeout per row
# grader_timeout_s = 150                # Grader timeout per row

branch and commit_sha are mutually exclusive. With branch, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository’s default branch.

See Config Files for the full TOML reference, including [env] and [secrets].

n is the number of attempts per selected dataset row. With limit = L and n = N, Osmosis runs up to L * N evaluations, or sampled_rows * n when sampling.

Status Lifecycle

Status	Description
pending	The run was submitted and is waiting to start.
running	The evaluation is executing against the selected rows.
finished	The evaluation completed successfully, and aggregate and sample results are available.
failed	The run ended because of an execution error. Check sample results and logs.
stopped	Someone stopped the run before it completed.

Monitoring

Platform Dashboard

Open the standalone Evaluation Runs page at platform.osmosis.ai to search and filter runs by status, dataset, model, and rollout. Each run page has a sidebar for status, sample progress, duration, pass rate, tokens used, submission details, dataset, model, and rollout, when those values are available. The run page has four tabs:

Overview: aggregate reward statistics and a pass@k view when the run has enough attempts.
Evaluation Samples: searchable, filterable sample outcomes with rewards, workflow messages, ground truth, grader metrics, and downloadable artifacts when present.
Configuration: the rollout entrypoint, pinned branch or commit, model, dataset, evaluation settings, resolved secret scopes, and environment keys.
Logs: lifecycle events and diagnostics for the run.

CLI Commands

# List evaluation runs for the current workspace repository
osmosis eval list
osmosis eval list --all

# Show details and the current result snapshot for one run
osmosis eval info my-eval-run

eval info reports run context, timing, progress, configuration, and aggregate results as they become available.

Managing Runs

Stopping a Run

osmosis eval stop my-eval-run

This requests a stop for a pending or running evaluation. The run moves to stopped after cleanup finishes. Pass --yes to skip the confirmation prompt.

Next Steps

Config Files

Reference for the evaluation TOML config.

Datasets

Upload and validate datasets for evaluation runs.

Training Runs

Submit a training run once your evaluation results look healthy.

Get started

Runs

Resources

Workspace

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Submitting an Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Next Steps

Config Files

Datasets

Training Runs

​Concepts

​Smoke Test or Formal Evaluation

​Evaluation Configuration vs Evaluation Run

​Dataset Input Modes

​Submitting an Evaluation Run

​Key Configuration Fields

​Status Lifecycle

​Monitoring

​Platform Dashboard

​CLI Commands

​Managing Runs

​Stopping a Run

​Next Steps

Config Files

Datasets

Training Runs

Concepts

Smoke Test or Formal Evaluation

Evaluation Configuration vs Evaluation Run

Dataset Input Modes

Submitting an Evaluation Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

Managing Runs

Stopping a Run

Next Steps