Training Runs

A training run improves a base model through reinforcement learning. You provide the rollout, grader, training config, and dataset; Osmosis runs the job on managed training infrastructure, pulls code from your synced workspace repository, and records metrics, samples, logs, and any checkpoints the run produces.

Concepts

Training Configuration vs Training Run

A Training Configuration is the recipe: it identifies the model, dataset, AgentWorkflow, and training controls. A Training Run is one execution of that configuration. Submit the configuration again to run another experiment.

Train and Validation Split

Osmosis shuffles the selected dataset and holds out about 20% of the rows for validation. The validation count is rounded down and capped at 10,000 rows, so larger datasets keep more than 80% for training. The split is not configurable.

Submitting a Training Run

Submit a training run with a TOML configuration file:

osmosis train submit configs/training/default.toml

Git Sync is the source of truth for your rollout code. The CLI reads config values from the local TOML file you pass, but rollout code comes from the synced workspace repository. Commit and push before submitting code changes. Set branch to use a pushed branch or commit_sha for a specific pushed revision; omit both to use the default branch.

Key Configuration Fields

[experiment]
rollout = "my-rollout"                  # Rollout directory name (under rollouts/)
entrypoint = "main.py"                  # Entrypoint file name
model_path = "Qwen/Qwen3.6-35B-A3B"     # Hugging Face model path
dataset = "my-dataset"                  # Dataset name
# branch = "my-feature"                 # Optional: use a pushed branch (default branch if omitted)
# commit_sha = "abc123..."              # Optional: pin to a specific synced commit

[training]
lr = 1e-6                               # Learning rate
total_epochs = 1                        # Number of training epochs
n_samples_per_prompt = 8                # Samples generated per prompt
rollout_batch_size = 32                 # Rollout batch size
agent_workflow_timeout_s = 450          # Agent rollout timeout per row
grader_timeout_s = 150                  # Grader timeout per row

[sampling]
rollout_temperature = 1.0               # Sampling temperature during rollouts
rollout_top_p = 1.0                     # Top-p sampling during rollouts

[checkpoints]
checkpoint_save_freq = 20               # Save a checkpoint every N rollout steps

branch and commit_sha are mutually exclusive. With branch, Osmosis resolves the branch head once at submission and stores that full commit SHA on the run. Omit both fields to use the repository’s default branch.

See Config Files for the full TOML reference. Submission validates the configuration against the shared schema and rejects unknown fields and out-of-range values; it does not check the configuration against the selected model or backend, so an unsupported combination can surface during provisioning or execution.

Status Lifecycle

Status	Description
pending	The run was accepted and is being prepared for scheduling.
queued	The run is ready and waiting for training capacity.
running	Training is in progress, and metrics and samples appear as they are produced.
finished	Training completed successfully. Final metrics and any produced checkpoints are available.
failed	The run ended because of an execution error. Check Rollout Samples and Logs for details.
stopped	Someone stopped the run before it completed.
killed	The training process was terminated while the run was being stopped or cleaned up.
crashed	The training process terminated unexpectedly (e.g. OOM, hardware failure).
unknown	The platform could not determine the run’s current state.

A failed or crashed run can still have checkpoints saved before it ended. A single rollout failure does not necessarily fail the run: failed samples are excluded from training updates. Use Rollout Samples to inspect sample outcomes and Logs to diagnose repeated or run-level failures.

Monitoring

Platform Dashboard

Open Training Runs at platform.osmosis.ai to search and filter runs by status, dataset, base model, and rollout. The list also supports opening, renaming, stopping, and deleting runs when the corresponding action is available. Each run page has a sidebar with status, rollout-step progress, duration, examples processed, submission and timing details, dataset, base model, and rollout. Its tabs are:

Overview: six charts for Training Reward, Validation Reward, Model Entropy, Response Length, Total Length, and Truncation Ratio. Charts appear when the corresponding metric is available and can be refreshed or downloaded as CSV.
Rollout Samples: browse samples by rollout step, search them, compare their reward distribution, and inspect each sample’s input and output. This is the first place to check individual rollout failures.
Checkpoints: plot training reward against saved checkpoint steps and list each checkpoint’s name, step, and reward. You can rename checkpoints, download their artifacts, open their model page, and deploy or undeploy them when inference deployment is available.
Configuration: inspect the rollout, entrypoint, model, dataset, pinned branch or commit, resolved secret scopes, environment keys, and stored training config for the run.
Logs: follow lifecycle events and diagnostics from submission through completion or cleanup, and download them when needed.

CLI Commands

# Show run details, checkpoints, and metrics
osmosis train info my-run

# Save the same information to a JSON file
osmosis train info my-run --output results/my-run.json

While a run is active, train info reports progress (current_step / total_steps) and the most recent reward. train list surfaces the same fields so you can scan runs at a glance.

LoRA Checkpoints

When the selected model and training backend support LoRA checkpoints, Osmosis saves them at the configured rollout-step cadence. A checkpoint captures the adapter weights at that point in training; a run is not guaranteed to produce a checkpoint if it ends before the first save or the selected training setup does not support one.

Managing Runs

Stopping a Run

osmosis train stop my-run

This requests a graceful stop. The run is marked stopped once the platform accepts the cancellation; resource cleanup continues afterward and its progress appears in the run’s Logs.

Next Steps

CLI Training

Submit and manage training runs from a workspace repository.

Datasets

Upload datasets for training.

Models

Manage base models and deploy trained LoRA models.

Get started

Runs

Resources

Workspace

Concepts

Training Configuration vs Training Run

Train and Validation Split

Submitting a Training Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

LoRA Checkpoints

Managing Runs

Stopping a Run

Next Steps

CLI Training

Datasets

Models

​Concepts

​Training Configuration vs Training Run

​Train and Validation Split

​Submitting a Training Run

​Key Configuration Fields

​Status Lifecycle

​Monitoring

​Platform Dashboard

​CLI Commands

​LoRA Checkpoints

​Managing Runs

​Stopping a Run

​Next Steps

CLI Training

Datasets

Models

Concepts

Training Configuration vs Training Run

Train and Validation Split

Submitting a Training Run

Key Configuration Fields

Status Lifecycle

Monitoring

Platform Dashboard

CLI Commands

LoRA Checkpoints

Managing Runs

Stopping a Run

Next Steps