configs/eval/; push your code before submitting because the platform clones the workspace repository identified by Git origin and runs the rollout server-side.
osmosis eval submit is also the recommended pre-flight before a training run — run it first to catch problems before committing GPU time.Quick Start
From inside your workspace directory:Evaluation Config
See Config Files for the full field reference.configs/eval/my-rollout.toml
When
[evaluation].limit is omitted, the platform evaluates a random 10% sample of the dataset (at least one row). Set limit to evaluate a fixed number of rows — the first N rows of the dataset, in order.How It Works
1
Resolve workspace and config
The CLI reads the evaluation TOML, resolves the workspace from the Git
origin remote, validates the config paths, and imports the configured rollout entrypoint so its backend can validate itself. That import is best effort: it is skipped with a warning when the local environment cannot satisfy the rollout’s declared dependencies, and the platform validates the entrypoint after installing them. It does not scan the module namespace to discover workflow or grader classes. See Files in a Rollout for the SDK entrypoint contract and the local-execution warning.2
Submit to the platform
The CLI submits the evaluation run request. The platform resolves the selected
branch or commit_sha once, clones that commit from the connected workspace repository, and prepares the evaluation environment.3
Validate the model
Before evaluating any rows, the platform runs a pre-flight check that confirms
[experiment].model_path is reachable with your configured credentials. If the model is unreachable — wrong name, missing or invalid API key, or provider rate limiting — the run fails early instead of consuming evaluation resources. Provide the model’s provider API key by registering it with osmosis secret set and listing it under [secrets].required (see Configuration Files).4
Run the rollout server-side
The platform starts your rollout and drives
AgentWorkflow.run(ctx) for each selected dataset row using [experiment].model_path as the evaluation policy. The row’s ground_truth is exposed to Grader.grade(ctx) as ctx.label, while optional row metadata is exposed as ctx.metadata; the grader runs when either a label or metadata is present and can use either or both.5
Aggregate results
The platform aggregates rewards, pass rates, and per-row results. Use
osmosis eval info <name> (or osmosis --json eval info <name>) to inspect them.Commands
See the Command Reference for the full flag list.
From Evaluation Run to Training Run
1
Submit an evaluation run
Run
osmosis eval submit configs/eval/my-rollout.toml. Use osmosis eval list and osmosis eval info <name> to track progress and inspect results.2
Iterate on rollout code
Push fixes to the workspace repository and re-submit. Use
branch for a feature branch or commit_sha to re-run against an older revision when comparing changes.3
Submit a training run
Once evaluation run results look healthy, run
osmosis train submit configs/training/my-rollout.toml. See Training.Download Run Outputs
Once a run has data, useosmosis eval download to pull metrics, trajectories, artifacts, and logs to your local disk instead of clicking through the web UI.
--overwrite, and downloads over 100 MiB require confirmation unless you pass --yes. The CLI retries failed files automatically and lists anything still missing so a second run picks it up. See the Command Reference for the full flag list.
osmosis eval info -o now points at the same run output root, and rich-mode metrics exports save to .osmosis/evals/<name>/metrics.json. Existing files under .osmosis/metrics/ are left untouched.Local Rubric Scoring
osmosis eval rubric is a local utility for scoring an existing JSONL conversation file with an LLM judge. It does not require a workspace directory or platform authentication, and it does not run a rollout. “Local” describes where the command runs: for each record the CLI sends the rubric and the scored assistant message — plus any original_input, ground_truth, or metadata fields on that record — to the provider behind --model, so review that provider’s privacy and retention terms before scoring sensitive data.
Next Steps
Config Files
Full reference for evaluation and training configuration files.
Git Sync
Push and sync rollout code before submitting evaluation runs or training runs.
Training
Submit and manage a training run once evaluation results look good.