Changelog

​

Breaking Changes

• The public MessageResult, GraderInitRequest, GraderInitResponse, RolloutDriver, and resolve_workspace_directory_from_cwd() APIs are removed. Use OperationResult, HttpRolloutDriver, and resolve_workspace_directory() instead; the two unused grader-init types have no replacement.
• The SDK no longer declares requests as a base dependency or tqdm in the rubric extra. Declare either package directly if your own code imports it.

​

Added

• osmosis eval run reaches Daytona, SkyPilot, and other cloud sandboxes through a managed cloudflared tunnel that starts automatically when the resolved sandbox cannot reach host loopback, or through a tunnel you run yourself with --advertise-url.
• Root --workspace <name> selects a platform workspace by name. Benchmark, dataset, model, secret, and run-inspection commands then work without a local repository, and eval submit and train submit accept an absolute config path when its Git repository matches the selected workspace.
• osmosis eval upload <run-name> resolves a completed run under .osmosis/evals/; explicit run directories still work.
• The eval installation extra is documented, and Python 3.14 is supported.

​

Changed

• Local osmosis eval run --dataset-file ... no longer loads platform credentials unless --upload is requested.
• Newly scaffolded rollouts depend on the stable osmosis-ai[server]>=0.3.0,<0.4 release line.
• Local evaluation output, retry, resume, and upload paths are printed relative to the directory the command was invoked from.

​

Fixed

• Local evaluation validates the model and rollout server before opening a tunnel, keeps slow non-streaming tunnel responses alive, accepts a registered Cloudflare connection when the host cannot probe the tunnel URL, and fails fast on an unreachable loopback endpoint instead of hanging.
• Local evaluation warns when the CLI and the rollout environment use different osmosis-ai versions.
• The harbor extra installs Harbor’s Daytona dependencies, so Daytona environments work without separate dependency setup.
• --secrets-file values override the process environment only for the local evaluation run and are restored on every exit path, and orphan cleanup rejects symlinked run directories.

​

Upgrade

​

Breaking Changes

• RolloutDriver.run now takes a single RolloutRunRequest argument. Update custom drivers and their callers to pass the request object.
• ExecutionBackend.max_concurrency and the import-time osmosis_ai.platform.auth.PLATFORM_URL are removed. Read concurrency capacity from the rollout server’s /health response, and the active platform URL from get_platform_url().
• Unused server-owned fields are removed from the public record types. Stop reading UploadInfo.s3_key and .upload_id, DatasetFile.df_stats and .organization_id, TrainingRunMetrics.training_run_id, EvalRunMetrics.eval_run_id, RolloutInfo.last_synced_at, and TrainingRunCheckpoints.training_run_id.

​

Added

• osmosis eval run executes a crash-safe local evaluation from the same TOML a managed run uses, through the new eval installation extra. It supports dataset slicing, resumable output, uv-managed rollout environments, LocalBackend and Harbor’s Docker environment, readable generated run names, OpenAI Responses routing, bounded admission, and orphan rollout-server cleanup.
• osmosis eval upload <run-dir> and eval run --upload import a completed local run into the platform. Both are server-authoritative and idempotent, and neither launches a managed evaluation.

​

Changed

• The CLI machine contract is standardized. --json and --plain never prompt, JSON errors use stable {code, message, details} envelopes on stderr, machine-readable warnings use JSON Lines, and non-finite values can no longer produce invalid JSON.
• Logins are platform-scoped and persist in the operating-system keyring across directories and environments. They survive HTTP 401 responses, and a non-production OSMOSIS_TOKEN is validated against OSMOSIS_TOKEN_PLATFORM_URL before any network access.

​

Fixed

• Mini SWE-agent benchmark credential validation matches the Platform, while the Cursor CLI harness-key requirement is preserved.
• osmosis dev server up prints the one-time API key the Platform returns, so the provisioned server can be used immediately.

​

Upgrade

​

Breaking Changes

• Every rollout now produces exactly one RolloutSample and reward through rollout-scoped URLs. Update custom contexts, graders, integrations, callbacks, and backend adapters to the single-sample contract.
• Runtime integrations use explicit server, strands, openai-agents, harbor, rubric, and parquet extras and feature-specific imports. AgentWorkflow.run() returns one message history, and the former loader, validator, and multi-trajectory helpers are removed.
• HarborBackend is now the container-native implementation previously called HarborBackendV2; the legacy backend, its constructor arguments, OsmosisInstalledAgent, and HarborAgentWorkflowContext are removed.
• Prompt and metadata datasets each use one uniform schema across every row, and benchmark detail output replaces required_secret_names with requires_judge_api_key.

​

Added

• osmosis benchmark now supports catalog discovery, submission, run inspection, logs, cancellation, and output downloads.
• The container-native Harbor backend adds installable workflow bundles, native agents, template and dataset modes, prewarming, diagnostics, artifacts, admission control, status polling, and cancellation.
• Train, evaluation, and benchmark submissions can resolve per-run secrets from a dotenv file, standard input, process environment, or hidden terminal prompt without saving them to the Platform secret store.
• osmosis quickstart guides users through authentication, workspace repository setup, cloning, billing checks, and a ready-to-paste agent prompt; matching onboarding APIs are available on OsmosisClient.

​

Reliability and Security

• Rollout bundles use content-addressed atomic caches, support flat, src/, and PEP 420 layouts, reject unsafe symlinks and cache layouts, and install a compatible uv builder through the Harbor extra.
• Local execution enforces controller deadlines, validates rewards, sanitizes optional callback telemetry, and preserves consistent terminal callback outcomes.
• Harbor preserves samples, diagnostics, and ATIF trajectories while preventing task-source symlinks and rollout credentials from entering archived trial data; secrets-file errors no longer echo secret-bearing input.

​

Upgrade

​

What changed

• osmosis quickstart now signs in, selects a workspace, waits for its repository connection, clones or reuses the workspace directory, checks billing, and produces a ready-to-paste agent prompt for training, evaluation, or benchmarks.
• OsmosisClient adds list_workspaces(), get_quickstart_status(), and complete_quickstart() for workspace-scoped onboarding integrations.
• RolloutSample.reward now rejects NaN, infinity, and non-numeric values at construction and assignment. A grader that assigns one of these values raises pydantic.ValidationError instead of silently reaching the controller as no reward at all. NumPy-like numeric scalars are normalized to float.
• LocalBackend counts time spent waiting on its concurrency limiter against the controller’s agent_timeout_sec, and applies an independent grader_timeout_sec around grading. Workflows that swallow CancelledError or block the event loop past their deadline are now reported as timeouts instead of returning a late success.
• osmosis eval submit, osmosis train submit, and osmosis benchmark submit accept --secrets-file lines written as export NAME=value and NAME="quoted value". The CLI reports a malformed line by source and line number only, so the secret text never reaches CI logs. The CLI also rejects names that do not pass Python’s identifier check up front.
• Rollout bundle builds are now content-addressed by project contents, interpreter ABI, and platform, and each build runs in an isolated directory that must produce exactly one wheel. src/ layouts (as scaffolded by uv init --lib) and PEP 420 namespace packages are supported alongside flat layouts, and the Harbor extra installs its compatible uv builder.
• Harbor rejects symlinks before task materialization and scrubs rollout controller credentials from retained trial files before they can be archived.

​

Who needs to act

• Upgrade to osmosis-ai>=0.3.0rc4 before using osmosis quickstart or relying on these rollout hardening fixes.
• Update any grader that could assign float("nan"), float("inf"), or a non-numeric object to ctx.set_reward. Osmosis now rejects these values. Return the intended numeric reward, or leave the reward unset to mean “not graded”.
• Update --secrets-file sources that relied on the previous parser accepting quotes as part of a value or an export prefix as part of a name. The old behavior silently submitted the wrong string; the new parser resolves both forms to the intended value.

​

What changed

• Prompt datasets require user_prompt plus ground_truth or its label alias; system_prompt remains optional.
• Metadata datasets require a non-empty metadata object on every row. Validation now checks every JSONL and CSV row and every Parquet metadata value.
• osmosis train submit, osmosis eval submit, and osmosis benchmark submit accept --secrets-file; values can also come from the process environment or an interactive prompt.
• Benchmark details replace required_secret_names with requires_judge_api_key.

​

Who needs to act

​

What changed

• Install optional features with the server, strands, openai-agents, harbor, rubric, or parquet extras; full installs all of them.
• Import server, Harbor, Strands, OpenAI Agents, and rubric features from their public submodules instead of the rollout or package root.
• AgentWorkflow.run() returns one message history as AgentWorkflowOutput, a bare message list, or None for the active sample-source fallback. Output metrics must be finite.
• The former HarborBackendV2 is now HarborBackend; the legacy backend and HarborAgentWorkflowContext were removed.
• The CLI adds benchmark catalog, submission, status, logs, cancellation, and output-download workflows.

​

Who needs to act

​

What changed

• GraderContext.samples is now GraderContext.sample, and set_sample_reward(sample_id, reward) is now set_reward(reward).
• Custom integrations register one source with set_sample_source() and read it with get_sample().
• RolloutSample.id and MultiTurnMode were removed. The rollout URL now supplies execution identity.
• Model and callback requests use rollout-scoped URLs. Integrations no longer attach per-call sample or rollout routing headers.
• Backends exchange sample.json and a single-value reward.json ({"reward": <float>}).
• Strands and OpenAI Agents integrations enforce one registered agent or session per workflow execution.

​

Who needs to act