Osmosis CLI Evaluation

​

Quick Start

osmosis eval run configs/eval/my-rollout.toml
osmosis eval run configs/eval/my-rollout.toml --dataset-file data/my-eval.jsonl
osmosis eval run configs/eval/my-rollout.toml --upload

osmosis eval upload <run-name>
osmosis eval upload .osmosis/evals/<run-name>/

osmosis --workspace <workspace-name> eval upload <run-name>

git push
osmosis eval submit configs/eval/my-rollout.toml

osmosis --workspace <workspace-name> eval submit /absolute/path/to/repository/configs/eval/my-rollout.toml

osmosis eval list
osmosis eval info <name>
osmosis eval stop <name>

​

Evaluation Config

[experiment]
rollout = "my-rollout"                # Rollout directory under rollouts/
entrypoint = "main.py"                # Entrypoint relative to the rollout directory
model_path = "openai/gpt-5-mini"      # LiteLLM-style model name for the evaluation policy
dataset = "my-platform-dataset"       # Platform dataset name from `osmosis dataset list`
# branch = "my-feature"               # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc1234"             # Optional: pin to a specific commit

[evaluation]
# Optional. Omit values to use defaults for the selected execution mode.
# limit = 200
# n = 1
# batch_size = 1
# pass_threshold = 1.0
# agent_workflow_timeout_s = 450
# grader_timeout_s = 150

# [env]
# LOG_LEVEL = "INFO"

[secrets]
# Required for eval configs. Use required = [] only when no secrets are needed.
required = ["OPENAI_API_KEY"]

​

How It Works

​

Local Execution and Upload

{"row_index": 0, "prompt": "What is 7 times 8?", "label": "56"}

• row_index is the row’s position among the selected rows, the same index index.jsonl uses. It matches the dataset-file offset unless you pass --rows.
• prompt is the row’s non-system message text, cut at 2,000 characters. A row without a prompt uses its metadata as sorted JSON instead.
• label is the row’s ground truth, cut at 500 characters. Rows without ground truth omit it.

• tokens is the total over every model call in the rollout.
• max_context_tokens is the largest single model call, counting its prompt plus completion tokens. Use it to size a model’s context window for training. Total tokens overstate the window you need, and per-call averages understate it.

[tool.uv.sources]
osmosis-ai = { path = "/path/to/osmosis-sdk-python", editable = true }

​

Source Digest Upgrades

• Choose a new run name.
• Pass --fresh to archive the previous directory under the same name and start a new run.

osmosis eval run configs/eval/<name>.toml --name <run-name> --fresh

​

Managed Execution

​

Inference Parameters

• Custom inference fields are forwarded. No inference-parameter allowlist applies; bridge controls remain blocked as described below. Set fields at the top level of the chat request or in the client’s extra_body; extra_body values win on conflicts. The OpenAI Python client merges extra_body into the top level of the request shallowly, so a nested object such as metadata in extra_body replaces the top-level object instead of merging with it.
• OpenRouter and OpenAI-compatible endpoints receive fields as request JSON. They are forwarded whether or not LiteLLM’s model catalog lists them.
• Native providers use LiteLLM translation without dropping parameters. An unsupported standard parameter raises an error instead of being silently removed. Translation follows LiteLLM and provider semantics, so it is not a byte-for-byte passthrough. For example, Anthropic metadata accepts only user_id, and native providers other than OpenAI, Azure, and Anthropic reject metadata.
• Provider rejections surface as errors. A provider 4xx response returns to your rollout with sanitized details. The bridge does not strip parameters and retry.

response = await client.chat.completions.create(
    model="osmosis-rollout",
    messages=messages,
    extra_body={
        "reasoning": {"effort": "low"},
        "provider": {"sort": "latency"},
    },
)

​

Commands

​

Retrying Failed Samples

osmosis eval retry <run-name>

osmosis eval run configs/eval/<name>.toml --name <run-name> --retry-failed --upload

osmosis eval run configs/eval/<name>.toml --name <run-name> --retry-failed &&
  osmosis --workspace <workspace-name> eval upload <run-name> --replace

​

From Evaluation Run to Training Run

​

Download Run Outputs

# Default: metrics + trajectories for the whole run
osmosis eval download my-eval-run

# Only trajectories and artifacts for selected rows
osmosis eval download my-eval-run --type trajectories,artifacts --rows 3,7,10-20

# Only the per-sample summary (status and reward), without trajectories
osmosis eval download my-eval-run --type summary

# Everything, into a custom root
osmosis eval download my-eval-run --type all -o ./eval-outputs/

.osmosis/evals/<run-name>/
├── metrics.json
├── summary.jsonl
├── trajectories/row_<row>_run_<n>.json
├── artifacts/row_<row>_run_<n>/...
└── logs.txt

​

Local Rubric Scoring

osmosis eval rubric -d conversations.jsonl \
  --rubric "Evaluate the assistant's helpfulness..." \
  --model openai/gpt-5-mini

​

Next Steps