Osmosis CLI Configuration Files

The Osmosis CLI uses TOML files for evaluation runs, training runs, and benchmark runs. Configs must live inside the workspace directory:

Config type	Required location	Command
Eval	`configs/eval/*.toml`	`osmosis eval submit`
Training	`configs/training/*.toml`	`osmosis train submit`
Benchmark	`configs/benchmark/*.toml`	`osmosis benchmark submit`

Required fields are shown un-commented. Optional fields are commented out in template files and can be omitted to use platform defaults.

Eval Config

Used by osmosis eval submit to submit an evaluation run. The platform clones the workspace repository identified by the origin remote and runs the rollout server-side against a platform dataset.

configs/eval/my-rollout.toml

[experiment]
rollout = "my-rollout"                         # Rollout directory under rollouts/
entrypoint = "main.py"                         # Entrypoint relative to rollout dir
model_path = "openai/gpt-5-mini"               # LiteLLM-style model name for the evaluation policy
dataset = "my-platform-dataset"                # Platform dataset name from `osmosis dataset list`
# branch = "my-feature"                        # Use a pushed branch (default branch if omitted)
# commit_sha = "abc1234"                       # Pin code to a specific commit

[evaluation]
# Optional. Omit values to use platform defaults.
# limit = 200                                  # First N rows; omit for random 10% sample
# n = 1                                        # Evaluation attempts per row
# batch_size = 1                               # Rows evaluated per batch
# pass_threshold = 1.0                         # Minimum passing score
# agent_workflow_timeout_s = 450               # Agent workflow timeout per row
# grader_timeout_s = 150                       # Grader timeout per row

# [env]
# LOG_LEVEL = "INFO"                           # Non-secret literal env var

[secrets]
# Required for eval configs. Default OpenAI eval models need this.
# Use required = [] when the evaluation needs no secret refs.
required = ["OPENAI_API_KEY"]

`[experiment]`

Field	Type	Required	Description
`rollout`	`str`	Yes	Rollout directory name under `rollouts/`
`entrypoint`	`str`	Yes	Python entrypoint relative to the rollout directory
`model_path`	`str`	Yes	LiteLLM-style model name for the evaluation policy (e.g. `openai/gpt-5-mini`)
`dataset`	`str`	Yes	Platform dataset name from `osmosis dataset list`
`branch`	`str`	No	Use the current head of this pushed branch. Mutually exclusive with `commit_sha`.
`commit_sha`	`str`	No	Pin code to a specific commit. Mutually exclusive with `branch`.

Omit both branch and commit_sha to use the latest synced commit on the repository’s default branch. Branch submissions resolve the branch head once, so the run remains pinned to the resulting full commit SHA.

`[evaluation]`

All fields are optional. Omit values to use platform defaults.

Field	Type	Range	Description
`limit`	`int`	> 0	Number of rows to evaluate (the first `N` rows). When omitted, the platform evaluates a random 10% sample of the dataset.
`n`	`int`	> 0	Number of evaluation attempts per row. It determines the attempts available for pass@k metrics, so the largest reported `k` cannot exceed `n`.
`batch_size`	`int`	> 0	Rows evaluated per batch
`pass_threshold`	`float`	—	Score at or above which a sample counts as passing; use the scale emitted by your grader
`agent_workflow_timeout_s`	`float`	`30–600`	Timeout for `AgentWorkflow.run()` per row
`grader_timeout_s`	`float`	`30–600`	Timeout for `Grader.grade()` per row

`[env]` and `[secrets]` (evaluation)

Optional [env] variables and a required [secrets] table for the evaluation run container. Eval configs must include [secrets] — use required = [] only when the evaluation needs no secret refs. See [env] and [secrets] below for the full ruleset.

Benchmark Config

Used by osmosis benchmark submit to run a benchmark already added to the current workspace. Benchmark configs describe the task selection, agent harnesses and models, and execution settings. They do not reference workspace rollout code.

configs/benchmark/terminal-bench-smoke.toml

[experiment]
benchmark = "terminal-bench-2-1"               # Benchmark key from `benchmark list`

# Omit [tasks] to run all tasks.
[tasks]
task_names = ["terminal-bench/git-multibranch"] # Explicit task names
# categories = ["software-engineering"]       # Union tasks from these categories
# task_set = "parity"                         # Benchmark-defined parity sample

[[agents]]
harness = "codex"

[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"             # Platform secret record name

# [agents.env]
# AGENT_MODE = "strict"                       # Literal env for this agent

[execution]
attempts_per_task = 1
max_concurrent_attempts = 4
timeout_multiplier = 1.0
max_retries = 0
# pass_threshold = 1.0
# Judge-enabled benchmarks only:
# judge_model = "openai/gpt-5.2"
# judge_api_key_secret = "OPENAI_API_KEY"

# Registry datasets whose verifier reads its own credentials:
# [verifier]
# required = ["MY_VLM_KEY"]                   # Platform secret record names

# [env]
# LOG_LEVEL = "info"                          # Literal env for every agent

`[experiment]` (benchmark)

Field	Type	Required	Description
`benchmark`	`str`	Yes	A benchmark already added to the current workspace, given as its key, name, or ID. All three are exact, case-sensitive matches

benchmark list shows both the key and the name; either one works here, as does the ID.

`[tasks]`

All fields are optional. Omit the section to run every task. task_names and categories are unioned when both are set. When task_set is set, its published sample is used instead of task_names or categories.

Field	Type	Description
`task_names`	`list[str]`	Explicit task names from the benchmark manifest
`categories`	`list[str]`	Include every task in the named categories
`task_set`	`str`	Named sample published by the benchmark; currently `parity`

For HLE, we recommend task_set = "parity". Omit [tasks] only when you intend to submit the full HLE benchmark.

`[[agents]]` and `[agents.model]`

Each run requires one to eight agents. Harness availability depends on the selected benchmark. Across managed benchmarks, supported harness values are claude-code, codex, cursor-cli, gemini-cli, mini-swe-agent, openhands, opencode, and terminus-2. The Platform rejects a harness that is unavailable for the selected benchmark. Every agent needs its own [[agents]] entry regardless: a benchmark that runs only its official scaffold rejects every harness, and one that merely allows a harness runs its official scaffold when you omit the field. benchmark info reports which of the three applies.

Agent field	Required	Description
`harness`	Benchmark-dependent	Agent harness offered by the selected benchmark; omit to run the benchmark’s official scaffold, where it has one
`harness_api_key_secret`	`cursor-cli` and `mini-swe-agent` only	`CURSOR_API_KEY` or `MSWEA_API_KEY` respectively, holding that harness’s credential; omit for every other harness
`model`	Yes	Provider, endpoint, or hosted model configuration
`env`	No	Literal environment variables for this agent

Model type	Required model fields	Use for
`provider`	`model`, `api_key_secret`	Provider models such as `openai/gpt-5.2`
`endpoint`	`base_url`, `model`, `api_key_secret`	OpenAI-compatible custom endpoints; `extra_headers` is optional
`hosted`	`base_model`, `lora_model_name`	A LoRA model you trained on Osmosis and deployed. `base_model` is the base it was trained on; `lora_model_name` is the LoRA model name

For provider and endpoint models, api_key_secret is the model’s Platform secret record name. harness_api_key_secret is separate and per-agent: it is required for cursor-cli and mini-swe-agent, and rejected for every other harness. Set it to CURSOR_API_KEY for cursor-cli and to MSWEA_API_KEY for mini-swe-agent. Those are the variables the harnesses read, and any other value is rejected at submit. Register the record with osmosis secret set, then reference its name in the agent. For example, a Cursor CLI agent can use:

[[agents]]
harness = "cursor-cli"
harness_api_key_secret = "CURSOR_API_KEY"

[agents.model]
type = "provider"
model = "openai/gpt-5.2"
api_key_secret = "OPENAI_API_KEY"

osmosis secret set CURSOR_API_KEY

To benchmark one of your own LoRA models instead of a provider model, use type = "hosted". Take both values from osmosis model list --type lora: base_model is the LoRA’s Base Model column and lora_model_name is its Name column.

[[agents]]
harness = "codex"

[agents.model]
type = "hosted"
base_model = "Qwen/Qwen3.6-35B-A3B"
lora_model_name = "deep-swe-agent"

Deploy the LoRA model with osmosis model deploy before submitting; a run against an undeployed one is rejected, as is a base_model that does not match what that LoRA model was trained on. Hosted agents need no api_key_secret, since Osmosis serves them.

Each agent’s effective environment combines top-level [env] with that agent’s [agents.env]. A provider or endpoint agent’s api_key_secret name cannot appear in that agent’s effective environment. Neither the judge_api_key_secret name nor any [verifier] secret name can appear in top-level [env] or any agent’s [agents.env].For a cursor-cli agent, do not also set CURSOR_API_KEY in either env table; for a mini-swe-agent agent, do not set MSWEA_API_KEY. The resolved harness secret owns that destination variable.Provider and endpoint api_key_secret fields also cannot reference runner-reserved names: DAYTONA_API_KEY, DAYTONA_API_URL, SKYPILOT_SERVICE_ACCOUNT_TOKEN, or SKYPILOT_API_SERVER_ENDPOINT. Those are Platform-managed sandbox plumbing, so store a model credential under a different record name.The platform reserves the _OSMOSIS_ prefix and rejects any key that starts with it in [env] or in any [agents.env].

All secret fields contain record names, never credential values. [agents.env] contains literal variables for one agent; top-level [env] applies to every agent, and agent-specific values override the same global key.

`[execution]`

Field	Type	Default	Range	Description
`attempts_per_task`	`int`	`1`	`1–10`	Attempts run for each selected task
`max_concurrent_attempts`	`int`	`4`	`1–64`	Maximum attempts running concurrently
`timeout_multiplier`	`float`	`1.0`	`> 0` and `<= 10`	Multiplier applied to benchmark task timeouts
`max_retries`	`int`	`0`	`0–5`	Retries for retryable infrastructure failures
`pass_threshold`	`float`	benchmark default	`0–1`	Score at or above which an attempt passes
`judge_model`	`str`	benchmark default	—	Optional judge model override for HLE and GDPVal
`judge_api_key_secret`	`str`	—	—	Platform secret record name for the judge or grader

A benchmark takes one of three shapes, shown in the LLM Judge row of benchmark info:

`LLM Judge` row	Meaning
`Required (default: <model>)`	HLE and GDPVal. `judge_api_key_secret` is required; `judge_model` is optional and falls back to the benchmark default
`API key only (pinned grader)`	BrowseComp. The adapter pins its own grader, so `judge_api_key_secret` is required and must be an OpenAI key, while `judge_model` is rejected
`–`	No judge. Both `judge_model` and `judge_api_key_secret` are rejected

`[secrets]`

A credential the run needs that no other field names — a registry dataset’s own API key, for instance. List the record name; the value is resolved at submit. The same section and resolution order apply to training and eval configs.

[secrets]
required = ["WEATHER_API_KEY"]

Resolution order per name, first hit wins:

--secrets-file <path>, one NAME=value per line with # comment lines skipped; - reads stdin
a variable of that name in the CLI process environment
an existing personal or workspace secret, resolved server-side
a hidden interactive prompt, only in a terminal and only for names still unresolved

Supplied locally (1, 2, or 4). The CLI reads the value and sends it with this submission’s TLS request. Osmosis records the name under the Run scope but does not add the value to the secret store or persistent run config, so you supply it again on every submit. Resolved server-side (3). The name is already a personal or workspace secret, so the platform looks the value up itself. Nothing to supply locally; a personal secret overrides a workspace secret with the same name. There is deliberately no --secret NAME=value flag: a value in argv lands in shell history, in ps output, and in any CI log running under set -x.

# Nothing to pass: the name is the env var name, so an exported value is found
osmosis benchmark submit configs/benchmark/custom.toml

# Many values from a dotenv file
osmosis benchmark submit configs/benchmark/custom.toml --secrets-file .env.local

# Training and eval take the same flag
osmosis train submit configs/training/my-run.toml --secrets-file .env.local

# Piped from a secret manager: secrets.env.tpl holds WEATHER_API_KEY={{ op://vault/weather/key }}, so no value lands on disk
op inject -i secrets.env.tpl | osmosis benchmark submit configs/benchmark/custom.toml --secrets-file - --yes

Stdin takes the same NAME=value lines a file does. Piping makes stdin non-interactive, so pass --yes to skip the submit confirmation. Outside a terminal, a name with no value fails immediately and lists every missing name at once, so CI reports the whole gap rather than one name per retry. Each --secrets-file line is dotenv-style NAME=value. The case-sensitive name must pass Python’s str.isidentifier() check. The CLI strips a leading export prefix and removes one matching pair of single or double quotes around the value. Blank lines and lines starting with # are ignored; inline comments, escape processing, and multiline values are not supported.

.env.run

# comment
WEATHER_API_KEY=plain-value
export ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY="sk-...with spaces"

A malformed line fails the submit. The CLI reports the source and line number only, and never echoes the line contents to stderr or the JSON error envelope, so a value cannot leak through CI logs.

`[verifier]`

Registry datasets declare credential templates the platform cannot enumerate, so a run names the Platform secret records supplying them here. Each record is delivered to the verifier under its own name, so name it for the variable the verifier reads. At most 16 names, each matching ^[A-Z][A-Z0-9_]*$.

[verifier]
required = ["MY_VLM_KEY"]

Managed benchmarks model their credentials in the catalog and take none, so submit rejects this section for them. A secret named here cannot also appear as a literal key in top-level [env] or any [agents.env], for the same reason as judge_api_key_secret.

Never put secret values in [env], [agents.env], or extra_headers. Benchmark configs use Osmosis fields and are not Harbor configuration files.

Training Config

Used by osmosis train submit to submit a training run.

configs/training/my-rollout.toml

[experiment]
rollout = "my-rollout"                         # Rollout directory under rollouts/
entrypoint = "main.py"                         # Entrypoint file name
model_path = "Qwen/Qwen3.6-35B-A3B"            # Supported base model
dataset = "my-dataset"                         # Platform dataset name
# branch = "my-feature"                        # Use a pushed branch (default branch if omitted)
# commit_sha = "abc1234"                       # Pin code to a commit

[training]
# lr = 1e-6                                    # Learning rate
# total_epochs = 1                             # Training epochs
# n_samples_per_prompt = 8                     # Rollout samples per prompt
# rollout_batch_size = 32                      # Rollout batch size
# max_prompt_length = 8192                     # Max prompt tokens
# max_response_length = 8192                   # Max response tokens
# agent_workflow_timeout_s = 450               # Agent timeout per row
# grader_timeout_s = 150                       # Grader timeout per row

[sampling]
# rollout_temperature = 1.0                    # Sampling temperature
# rollout_top_p = 1.0                          # Top-p sampling

[checkpoints]
# eval_interval = 10                           # Evaluate every N rollouts
# checkpoint_save_freq = 20                    # Save checkpoint every N rollouts

# [advanced]
# Available only to explicitly enabled accounts; standard configs must omit it.

# [env]
# LOG_LEVEL = "INFO"                           # Non-secret literal env var

# [secrets]
# required = ["OPENAI_API_KEY"]                # Optional in training; if set, must include `required`

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

`[experiment]`

Field	Type	Required	Description
`rollout`	`str`	Yes	Rollout directory name under `rollouts/`
`entrypoint`	`str`	Yes	Python entrypoint file name, usually `main.py`
`model_path`	`str`	Yes	Supported base model path
`dataset`	`str`	Yes	Dataset name from `osmosis dataset list`
`branch`	`str`	No	Use the current head of this pushed branch. Mutually exclusive with `commit_sha`.
`commit_sha`	`str`	No	Git commit SHA to fetch from the workspace repository. Mutually exclusive with `branch`.

`[training]`

Field	Type	Default	Range	Description
`lr`	`float`	platform default	—	Learning rate
`total_epochs`	`int`	platform default	>= 1	Number of training epochs
`n_samples_per_prompt`	`int`	platform default	`1–12`	Rollout samples generated per prompt
`rollout_batch_size`	`int`	platform default	`1–64`	Prompts processed per rollout batch
`max_prompt_length`	`int`	platform default	`1–8192`	Maximum prompt tokens
`max_response_length`	`int`	platform default	`1–10240`	Maximum response tokens
`agent_workflow_timeout_s`	number	platform default	`30–600`	Agent rollout timeout per row
`grader_timeout_s`	number	platform default	`30–600`	Grader timeout per row

`[sampling]`

Field	Type	Default	Range	Description
`rollout_temperature`	number	platform default	>= 0	Sampling temperature during rollouts
`rollout_top_p`	number	platform default	>= 0	Top-p sampling threshold

`[checkpoints]`

Field	Type	Default	Range	Description
`eval_interval`	`int`	platform default	>= 1	Evaluate every N rollout steps
`checkpoint_save_freq`	`int`	platform default	>= 1	Save a LoRA checkpoint every N rollout steps

The platform enforces these ranges at submission. It does not check values against the selected model or backend, so an unsupported combination can surface during provisioning or execution.

`[advanced]`

Backend-specific fields for accounts explicitly enabled by Osmosis. Standard accounts and standard configs must omit [advanced]; the platform rejects it when the account is not enabled.

`[env]` and `[secrets]`

Use these sections to inject environment variables into the rollout container during training runs or evaluation runs. The same shape applies to both training and evaluation configs.

Section	Values	Use for
`[env]`	Literal strings stored in the config file	Non-secret configuration
`[secrets].required`	List of platform `environment_secret` record names	API keys and private credentials

[env]
LOG_LEVEL = "INFO"

[secrets]
required = ["OPENAI_API_KEY", "DATABASE_URL"]

Rules:

[env] keys must match ^[A-Z_][A-Z0-9_]*$; [secrets].required names must match ^[A-Z][A-Z0-9_]*$.
The same name cannot appear in both [env] and [secrets].required.
[env] keys starting with _OSMOSIS_ are reserved by the platform and cannot be used.
Existing personal or workspace secret records are resolved server-side, so their values do not appear in the local config or CLI request. Values supplied locally through --secrets-file, the process environment, or a hidden TTY prompt travel in that submission’s TLS request but are not added to the Osmosis secret store or persistent run config.
Eval configs must include [secrets]. Use required = [] only when the evaluation needs no secret refs.
Training configs may omit [secrets]. If you include the table, it must define required.

Secrets are scoped. A workspace secret is shared across the workspace; a personal secret is private to you and overrides the workspace secret of the same name at run time. Register secrets with osmosis secret set before submitting a run that references them.

Start with only [experiment] (plus [secrets] for eval configs) and let the platform use training defaults. Add optional fields only when you need to tune a run.

Get started

Runs

Workspace

Reference

Osmosis CLI Configuration Files

Eval Config

`[experiment]`

`[evaluation]`

`[env]` and `[secrets]` (evaluation)

Benchmark Config

`[experiment]` (benchmark)

`[tasks]`

`[[agents]]` and `[agents.model]`

`[execution]`

`[secrets]`

`[verifier]`

Training Config

`[experiment]`

`[training]`

`[sampling]`

`[checkpoints]`

`[advanced]`

`[env]` and `[secrets]`

​Eval Config

​[experiment]

​[evaluation]

​[env] and [secrets] (evaluation)

​Benchmark Config

​[experiment] (benchmark)

​[tasks]

​[[agents]] and [agents.model]

​[execution]

​[secrets]

​[verifier]

​Training Config

​[experiment]

​[training]

​[sampling]

​[checkpoints]

​[advanced]

​[env] and [secrets]

Eval Config

`[experiment]`

`[evaluation]`

`[env]` and `[secrets]` (evaluation)

Benchmark Config

`[experiment]` (benchmark)

`[tasks]`

`[[agents]]` and `[agents.model]`

`[execution]`

`[secrets]`

`[verifier]`

Training Config

`[experiment]`

`[training]`

`[sampling]`

`[checkpoints]`

`[advanced]`

`[env]` and `[secrets]`