---
name: osmosis
description: Use when building, evaluating, and training LLM agent workflows.
  Reach for this skill when agents need to create rollouts (AgentWorkflow +
  Grader), run local or managed evaluations, submit training runs, or manage
  datasets and models on the Osmosis platform.
metadata:
  mintlify-proj: osmosis
  version: "1.0"
---

# Osmosis AI Skill Reference

## Product Summary

Osmosis is a platform for building, evaluating, and training LLM agent workflows. It combines a Python SDK (`osmosis-ai`), a CLI (`osmosis`), and a web platform to define agent behavior, score results with graders, run evaluations against datasets, and train specialized models using reinforcement learning.

**Key files and commands:**
- Rollout entrypoint: `rollouts/<name>/main.py` (defines `AgentWorkflow` and `Grader` classes)
- Evaluation config: `configs/eval/<name>.toml` (specifies rollout, model, dataset, and evaluation settings)
- Training config: `configs/training/<name>.toml` (specifies rollout, base model, dataset, and training hyperparameters)
- CLI: `osmosis eval run`, `osmosis eval submit`, `osmosis train submit`, `osmosis dataset list`, `osmosis secret set`
- Platform: https://platform.osmosis.ai (web dashboard for runs, datasets, models, and results)

**Primary docs:** https://docs.osmosis.ai

## When to Use

Use Osmosis when:
- An agent needs to define behavior in Python (`AgentWorkflow.run()`) and score results (`Grader.grade()`)
- You need to evaluate a rollout against a dataset locally or on managed infrastructure
- You want to train a base model on a task using RL with your rollout and grader
- You're managing datasets, secrets, or trained LoRA models for agent training
- You need to benchmark agents against published task suites
- You're integrating with Strands Agents or OpenAI Agents SDK for tool use and message handling

Do not use Osmosis for: direct LLM API calls outside a rollout context, non-agent tasks, or tasks that don't require evaluation or training.

## Quick Reference

### Core Abstractions

| Abstraction | Purpose | Where it lives |
| --- | --- | --- |
| `AgentWorkflow` | Defines agent behavior: receives prompt, calls model, uses tools, produces sample | `rollouts/<name>/main.py` |
| `Grader` | Scores the sample: receives sample + label, assigns numerical reward | `rollouts/<name>/main.py` |
| `OsmosisRolloutModel` | Placeholder model that resolves to current policy at runtime | Passed to agent framework (Strands or OpenAI Agents) |
| `LocalBackend` | Runs workflow in-process; fastest for iteration | Constructed in entrypoint |
| `HarborBackend` | Runs workflow in isolated container; template or dataset mode | Constructed in entrypoint |

### Essential CLI Commands

| Command | Use for |
| --- | --- |
| `osmosis eval run <config>.toml` | Execute evaluation locally; iterate fast |
| `osmosis eval run <config>.toml --upload` | Publish completed local evaluation to platform |
| `osmosis eval submit <config>.toml` | Submit full-size managed evaluation (required gate before training) |
| `osmosis eval info <name>` | Inspect evaluation results and metrics |
| `osmosis train submit <config>.toml` | Submit training run after evaluation passes |
| `osmosis train info <name>` | Monitor training progress and checkpoints |
| `osmosis dataset list` | List available platform datasets |
| `osmosis dataset upload <file>` | Upload JSONL, CSV, or Parquet dataset |
| `osmosis secret set <NAME>` | Register API key or credential for runs |
| `osmosis quickstart` | Onboard: authenticate, clone workspace, set up billing |

### Config File Locations

```
repository/
├── rollouts/
│   └── my-rollout/
│       ├── main.py                    # AgentWorkflow + Grader + backend wiring
│       └── pyproject.toml             # Rollout dependencies
├── configs/
│   ├── eval/
│   │   └── my-rollout.toml            # Evaluation config
│   └── training/
│       └── my-rollout.toml            # Training config
└── data/
    └── test.jsonl                     # Local test dataset
```

### Evaluation Config Template

```toml
[experiment]
rollout = "my-rollout"
entrypoint = "main.py"
model_path = "openai/gpt-5-mini"
dataset = "my-platform-dataset"

[evaluation]
limit = 100                            # First N rows (omit for all local, 10% managed)
n = 1                                  # Attempts per row
batch_size = 1
pass_threshold = 1.0

[secrets]
required = ["OPENAI_API_KEY"]
```

### Training Config Template

```toml
[experiment]
rollout = "my-rollout"
entrypoint = "main.py"
model_path = "Qwen/Qwen3.6-35B-A3B"
dataset = "my-platform-dataset"

[training]
lr = 1e-5
total_epochs = 1
n_samples_per_prompt = 8
rollout_batch_size = 32

[sampling]
rollout_temperature = 1.0
rollout_top_p = 1.0

[checkpoints]
eval_interval = 10
checkpoint_save_freq = 20
```

## Decision Guidance

### When to Use LocalBackend vs HarborBackend

| Scenario | Use LocalBackend | Use HarborBackend |
| --- | --- | --- |
| Developing and debugging rollout code | ✓ | — |
| Fast iteration with direct exceptions | ✓ | — |
| Workflow needs isolated filesystem or environment | — | ✓ (template mode) |
| Existing Harbor tasks with instructions and verifiers | — | ✓ (dataset mode) |
| Production evaluation or training runs | ✓ or ✓ | Both supported |

### When to Use Strands vs OpenAI Agents Integration

| Scenario | Use Strands | Use OpenAI Agents |
| --- | --- | --- |
| Tools are already `@tool` functions | ✓ | — |
| Migrating from `strands.Agent` | ✓ | — |
| Using `Runner.run` and sessions | — | ✓ |
| Migrating from OpenAI Agents SDK | — | ✓ |
| Need Strands message traces | ✓ | — |
| Need OpenAI Responses API sessions | — | ✓ |

### When to Use Local Eval vs Managed Eval vs Training

| Goal | Use `eval run` | Use `eval submit` | Use `train submit` |
| --- | --- | --- | --- |
| Smoke test rollout code | ✓ | — | — |
| Iterate on workflow/grader | ✓ | — | — |
| Full-size evaluation gate | — | ✓ | — |
| Validate before training | — | ✓ | — |
| Improve model on task | — | — | ✓ |

## Workflow

### 1. Set Up Workspace and Authenticate

```bash
osmosis quickstart
# Follows wizard: authenticate, clone repository, set up billing
cd <workspace-directory>
```

### 2. Create or Inspect Rollout

```bash
# List available templates
osmosis template list

# Apply a starter (Strands or OpenAI Agents)
osmosis template apply multiply-local-strands

# Or write rollouts/<name>/main.py manually with AgentWorkflow + Grader
```

### 3. Prepare Dataset

```bash
# Upload dataset to platform
osmosis dataset upload data/my-dataset.jsonl

# Or use an existing platform dataset
osmosis dataset list
```

### 4. Create Evaluation Config

Write `configs/eval/my-rollout.toml` with rollout name, entrypoint, model, and dataset. Register secrets:

```bash
osmosis secret set OPENAI_API_KEY
```

### 5. Run Local Evaluation

```bash
osmosis eval run configs/eval/my-rollout.toml
# Iterates locally; check logs and results in .osmosis/evals/<run-name>/
```

### 6. Iterate Rollout Code

Edit `rollouts/<name>/main.py`, re-run local eval, push to Git when settled:

```bash
git add rollouts/ configs/
git commit -m "Improve workflow"
git push
```

### 7. Submit Managed Evaluation

```bash
osmosis eval submit configs/eval/my-rollout.toml
osmosis eval info <run-name>
# Wait for completion; check results on platform
```

### 8. Create Training Config

Write `configs/training/my-rollout.toml` with rollout, base model, dataset, and hyperparameters.

### 9. Submit Training Run

```bash
osmosis train submit configs/training/my-rollout.toml
osmosis train info <run-name>
# Monitor progress; inspect checkpoints when ready
```

### 10. Deploy or Export Trained Model

```bash
osmosis model list --type lora
osmosis model deploy <lora-name>
# Use in benchmarks or export for inference
```

## Common Gotchas

- **Model calls must route through Osmosis integrations.** Do not call OpenAI SDK or LiteLLM directly with a hard-coded model in `AgentWorkflow.run()`. Use `OsmosisRolloutModel` with Strands or OpenAI Agents SDK. Direct calls bypass the rollout context and break training.

- **Construct agents inside `run()`, not at module import time.** The `RolloutContext` is only available during execution. Constructing `OsmosisStrandsAgent` or `OsmosisAgent` at import time raises `RuntimeError`.

- **Secrets are names, not values.** Never put API keys in TOML files or command arguments. Register with `osmosis secret set <NAME>`, then reference the name in `[secrets].required`. Values are resolved from environment, `--secrets-file`, or interactive prompt.

- **Evaluation is a gate before training.** A managed `eval submit` is required before `train submit`. Local eval results do not replace this gate; upload with `--upload` to make them visible but they still don't bypass the requirement.

- **Push code before submitting managed runs.** Both `eval submit` and `train submit` clone from the repository. Unpushed changes are not included. Use `branch` or `commit_sha` in config to pin a specific revision.

- **Grader must set reward or skip sample.** Call `ctx.set_reward(float)` or leave it unset (do not call the method) to skip grading. Returning `None` from `grade()` without calling `set_reward` leaves the sample ungraded.

- **Reward must be finite.** `NaN`, `inf`, and `-inf` are rejected. Return a float between 0.0 and 1.0 or any finite value.

- **Artifacts directory can be `None`.** Check `if ctx.artifacts_dir:` before writing files in workflow or grader. Unguarded writes raise and fail the rollout.

- **Eval config must include `[secrets]`.** Use `required = []` only when no secrets are needed. Training configs may omit `[secrets]` entirely.

- **Workspace billing must be set up.** Training runs, managed evaluations, and benchmarks require a valid payment method. Self-serve workspaces need an owner or admin to add one.

- **Secrets cannot appear in `[env]`.** A secret name cannot also be a literal environment variable. Use `[secrets].required` for credentials and `[env]` for non-secret config.

## Verification Checklist

Before submitting an evaluation or training run:

- [ ] Rollout entrypoint imports without errors: `python -c "from rollouts.my_rollout.main import app"`
- [ ] `AgentWorkflow.run()` uses `OsmosisRolloutModel` or agent integration, not direct SDK calls
- [ ] `Grader.grade()` calls `ctx.set_reward()` with a finite float or leaves it unset
- [ ] Evaluation config has `[experiment]` with rollout, entrypoint, model_path, dataset
- [ ] Evaluation config has `[secrets].required` listing all API keys (or `required = []` if none)
- [ ] All secrets registered: `osmosis secret set <NAME>` for each name in config
- [ ] Dataset exists: `osmosis dataset list | grep <dataset-name>`
- [ ] Code is pushed: `git push` before `eval submit` or `train submit`
- [ ] Local eval passes: `osmosis eval run configs/eval/my-rollout.toml` completes without errors
- [ ] Managed eval passes: `osmosis eval submit` completes and shows healthy results before training
- [ ] Training config has `[experiment]` with rollout, entrypoint, model_path, dataset
- [ ] Base model is supported: `osmosis model list --type base | grep <model-path>`
- [ ] Workspace billing is active: check Platform Settings → Billing

## Resources

**Comprehensive page listing:** https://docs.osmosis.ai/llms.txt

**Critical documentation:**
1. [SDK Overview](https://docs.osmosis.ai/sdk/overview) — Core abstractions, training loop, file layout, backend choice
2. [CLI Evaluation](https://docs.osmosis.ai/cli/evaluation) — Local and managed eval workflows, config reference, retry logic
3. [Config Files](https://docs.osmosis.ai/cli/config-files) — Complete TOML schema for eval, training, and benchmark configs

---

> For additional documentation and navigation, see: https://docs.osmosis.ai/llms.txt