Changelog

​

Breaking Changes

• eval upload and eval run --upload require a platform with archive import support. There is no per-file upload fallback, so upgrade the platform before the CLI.
• Direct integrations that call OsmosisClient.start_eval_run_import() must pass archive= instead of files=. Upload files.jsonl and run.zip, then start and poll the extraction with start_eval_run_import_extraction() and get_eval_run_import_extraction() until it reaches ready before finalizing. See the migration guide.
• Eval-import results require upload_mode, list archive upload slots in files, and allow a slot’s sha256 to be None. Update model constructors and result handling.

​

Changed

• Local eval uploads stream a ZIP64 archive one part at a time without writing it to disk. This replaces per-file requests, and retries skip upload slots the platform already has.

​

Fixed

• Strands rollouts keep earlier assistant reasoning in follow-up requests, live previews, and ATIF trajectories instead of dropping reasoning_content.

​

Upgrade

​

Breaking Changes

• Named local eval runs created with 0.3.8 or earlier refuse resume and --retry-failed after the source digest format changes, even if rollout files are unchanged. Finish interrupted runs with their original SDK, choose a new name, or use --fresh to archive and restart. Old manifests and results are preserved, and completed runs remain uploadable. See the migration guide.

​

Added

• agent_started_at distinguishes setup failures from attempts whose agent started. Upgrade both the caller and rollout server before relying on this field.
• Built-in OpenSandbox environments accept a positive whole-number OPENSANDBOX_REQUEST_TIMEOUT_SEC default for client requests; explicit timeout settings take precedence.
• Local eval uploads include dataset_rows.jsonl for prompt and ground-truth search and ground-truth sorting in Evaluation Samples. Old completed runs can still be uploaded without it.
• eval download supports paged manifests with progress reporting and --type summary to download only the per-sample summary. Dataset-row uploads and explicit summary downloads require matching platform support.

​

Changed

• The strands, openai-agents, harbor, rubric, and eval extras require LiteLLM >=1.103.1,<2.0.0 excluding 1.104.0rc1, and filelock >=4.2.1, for security fixes.
• The strands extra requires Strands Agents >=1.48.0,<2.0.0. The server, strands, harbor, and eval extras require OpenTelemetry API >=1.45.1,<2.0.0. The full extra includes all updated requirements.

​

Fixed

• Known credentials are redacted more consistently from CLI output, debug traces, platform errors, provider diagnostics, and local eval logs, including escaped and multiline values. Redaction only covers known secret values; datasets, trajectories, and artifacts still need review before sharing.
• Malformed provider responses use sanitized error handling. HTTP 426 responses with a non-string status show the standard upgrade message. Rubric dataset loading skips non-object messages and reports a line-numbered error when no usable assistant response remains.

​

Upgrade

​

Breaking Changes

• RolloutInfo.is_active, the rollout list Active column, and the is_active JSON field are removed. Remove reads of this always-true flag and any RolloutInfo(is_active=...) constructor argument. JSON now includes description and last_synced_at as descriptive and sync metadata. See the migration guide.

​

Added

• Harbor prewarm supports auto, full, healthcheck, and off modes, discovers dataset tasks when task IDs are omitted, reports progress, and limits concurrency when OPENSANDBOX_MAX_SANDBOXES is set to a positive integer. Isolated task failures allow startup with those tasks unavailable; systemic failures stop startup. See the Harbor backend.
• Train, eval, and benchmark submit confirmations show the target workspace or its connected repository first. The same target appears in --json confirmation details. See workspace scope.
• Base model info shows input, cached-input, and output prices, retaining up to four decimal places for prices below $0.10. JSON always includes all three price fields. Unavailable prices appear as – in the terminal and null in JSON.

​

Changed

• The harbor extra supports Harbor >=0.20.0,<0.25. Native OpenCode reuses compatible runtimes prebaked under /opt/osmosis-agent/bin instead of reinstalling them for each trial.
• The strands, openai-agents, harbor, rubric, and eval extras require aiohttp>=3.14.4 and multidict>=6.9.1. The Strands and OpenAI Agents integrations also require MCP >=1.30.0, excluding 2.0.* and 2.1.*.
• The base installation requires certifi>=2026.10.19. The harbor and eval extras require uv>=0.13.4,<0.14. Both updates remove the affected emSign root from their separate CA trust stores.

​

Fixed

• train info --output keeps an existing file unless you pass --overwrite. Metrics exports are written atomically; a failed save reports save_warning while the command still succeeds.
• Local eval no longer waits for a nonexistent trajectory after a rollout returns no sample, and its live pass rate excludes skipped samples. A second Ctrl-C skips long shutdown waits and gives the rollout server five seconds before force-stopping it, while still stopping the tunnel and listener. Sandbox cleanup may remain incomplete.
• New training scaffolds omit unsupported eval_interval examples and point to osmosis model list --type base for supported models.

​

Upgrade

​

Added

• The harbor extra now includes OpenSandbox support. HarborBackend applies protocol and managed proxy defaults for built-in OpenSandbox environments. Explicit environment kwargs that you pass still take precedence. See the Harbor backend.

​

Changed

• The strands, openai-agents, harbor, rubric, and eval extras, including full, now require fsspec>=2026.6.0. This version fixes a code-execution issue in fsspec’s ReferenceFileSystem. Update conflicting application pins, regenerate lockfiles, and rebuild affected runtime images.

​

Fixed

• Harbor tasks without a Dockerfile now skip SDK pre-install patching instead of failing. Tasks that use prebuilt images run with the default bundle setup.
• RolloutClient retries result reads that return HTTP 502, 503, or 504 up to six times within the polling lease.
• New rollout scaffolds and the Harbor source gateway keep connections alive longer than the platform load balancer’s idle timeout.

​

Upgrade

​

Breaking Changes

• POST /drain, RolloutClient.drain(), and the drain request and response models are removed. Use RolloutClient.health() and wait_idle(process_id=...) to observe completion, and send SIGTERM to shut down the rollout server. wait_idle() does not block new admissions or confirm external uploads, so keep your submitted rollout IDs for evidence verification.

​

Added

• Rollout servers call ExecutionBackend.shutdown() during shutdown to clean up backend resources, including Harbor trials and artifacts. Set your host’s termination grace period to cover the server’s bounded shutdown stages and Harbor’s additional cleanup wait.
• Hosts can enable bounded live trajectory previews for supported local sources, native Harbor agents, and bundled workflows. Previews do not change final results or grading. Bundled workflows need a preview-capable SDK inside the sandbox.

​

Changed

• The SDK supports Harbor >=0.20.0,<0.24 and keeps published harbor-v1 image identities and Mini-SWE endpoints. Before you upgrade to Harbor 0.23, review its stricter input validation, finite JSON rewards, Mini-SWE’s managed Python 3.12 setup, and Terminus ATIF 1.8 consumers. See the Harbor backend.
• Extras that include them now require LiteLLM >=1.94.3,<2.0.0 (excluding 1.95.0.*, 1.96.0.*, and 1.96.1.*), urllib3 >=2.8.0, and PyJWT >=2.15.1 for security fixes. Update conflicting application pins and regenerate your lockfiles.

​

Fixed

• Existing Harbor subclasses keep their original build_trial_config, build_agent_config, and harness_agent_config signatures. Subclasses no longer fail at startup with an unexpected preview_path keyword.
• Unexpected execution task crashes now produce a terminal internal_error failure. Clients no longer poll until the lease expires.
• Native evidence redaction stays linear on long identifiers. It no longer stalls health checks and result polling.

​

Upgrade

​

Breaking Changes

• The osmosis dev server and osmosis images command groups are removed from the public CLI. The related dev-server and image-build methods, models, and enums on OsmosisClient are also removed.
• Harbor rollout IDs can no longer contain colons or control characters. Use portable IDs so retained evidence paths work on every file system.

​

Added

• Rollout servers gained optional bearer authentication, and RolloutClient added health(), wait_idle(), and drain() for process identity and lifecycle control. The drain() API was removed in 0.3.6; use the shutdown guidance in that release entry when upgrading.
• Harbor source gateways run pinned GitHub tasks with immutable images and validated HarborGatewayConfig settings. SourceImageManifest and SourceImageBuildResult validate image bindings and task selections.
• Harbor retains sanitized native results and logs with checksum inventories. Native-log exports exclude OpenCode databases and snapshots.
• osmosis_ai.cli.main.create_app() and run_cli() let you build a downstream CLI on the public command tree, root options, and output handling, then add your own commands.

​

Changed

• JSON error envelopes resolve command from the invoked command tree. Unknown subcommands, such as dataset lst, are kept, and --workspace values are no longer reported as the command.

​

Fixed

• Local eval and the OpenAI Agents and Strands integrations preserve custom inference parameters, including OpenRouter reasoning and provider fields. Bridge-control overrides are rejected.

​

Upgrade

​

Added

• osmosis eval retry <name|id> retries failed and skipped samples in a hosted evaluation and keeps results that were already graded. To replace uploaded results after a local retry, run osmosis eval upload <run-name> --replace or osmosis eval run <config_path> --name <run-name> --retry-failed --upload.
• osmosis model info and OsmosisClient.get_base_model() accept base model names or Hugging Face paths. They report parameter counts, context windows, and inference pricing when available.
• HarborBackend(agent="opencode") runs native OpenCode through the rollout chat-completions endpoint, with compaction and pruning disabled. Native agent options, including version pins, also apply during prewarm. See the Harbor backend.

​

Changed

• The SDK now requires anyio>=4.14.2 for security fixes. Update older AnyIO pins in your CLI and rollout environments before upgrading.

​

Fixed

• Harbor artifact archiving no longer blocks the event loop or stalls result polling on slow storage. Rollouts still wait for archiving to finish before they return results.
• RolloutClient retries transient result-polling disconnects and read timeouts up to twice, within the last confirmed polling lease deadline. Remove any custom retry wrappers around result polling to avoid nested retries.
• Identifiable sandbox provider failures now report HTTP_ERROR. Task-level API errors and generic Harbor API errors without a structured status still report AGENT_ERROR.
• CLI failures show actionable platform and login explanations with consistent HTTP error categories. Skipped rollout preflight warnings include installation guidance for the CLI’s environment.

​

Upgrade

​

Breaking Changes

• Upgrade rollout callers and servers together. Replace HttpRolloutDriver and callback models with RolloutClient; each request needs a unique rollout_id and an explicit llm_api_key when its chat endpoint requires authentication.
• Custom backends return ExecutionOutcome from execute(request) and publish progress with await RolloutContext.set_status(). Move local LLM bridge imports from osmosis_ai.rollout.controller to osmosis_ai.eval.local.
• Local evaluation’s protocol fingerprint is now 0.4. Runs recorded under the previous protocol need a new run name, or the previous SDK to resume.
• Managed SkyPilot placement is removed. Select EnvironmentType.DAYTONA with Daytona credentials; HARBOR_SKYPILOT_CONTEXT is no longer read. Omitting environment_config still selects Docker.

​

Added

• RolloutClient handles lease renewal, HTTP 429 admission retries, cancellation, and optional grade=False. await run_rollout_async() returns an awaitable RolloutHandle with live status and milestone waits.
• osmosis eval upload and osmosis eval run --upload include logs.txt in the platform Logs tab. Local logging and upload redact known ambient provider and platform credentials of at least eight characters available in the current process environment; review logs before sharing because redaction does not detect every possible secret.

​

Changed

• Built-in Daytona environments with delete=True default to stopping after 60 minutes of Daytona-observed inactivity and immediate deletion. Increase auto_stop_interval_mins for longer periods without provider-visible activity, or set it to 0 to disable auto-stop. See Daytona lifecycle defaults.
• Supported dependency ranges expand to Harbor >=0.20.0,<0.23 and OpenAI Agents >=0.18.1,<0.21. When upgrading Harbor, remove top-level trajectory.json inputs from SDK workflow task directories and check Docker host nftables support before restricted-network trials. Generated rollout trajectory artifacts remain supported.

​

Fixed

• Polling preserves completed rewards and failure details, bounds admission and cancellation cleanup, and leaves local evaluation work without a recorded terminal result pending for resume.
• Login falls back to an owner-only credentials file when the system keyring is unavailable. OSMOSIS_TOKEN_STORE=keyring requires the keyring; file selects file storage explicitly. See credential storage.
• osmosis train info lists checkpoints while a training run is still running.

​

Upgrade

​

What changed

• RolloutClient.run_rollout_async() now returns an awaitable RolloutHandle instead of asyncio.Task. The handle exposes status, latest_result, done(), cancel(), and the milestone waits wait_for_running(), wait_for_grading(), and wait_for_completion(). A milestone wait also finishes when the phase has already passed or the rollout terminates.
• Result long polling returns on status changes, including grading milestones from the Local and Harbor backends, without waiting for the full polling timeout.
• Custom backends publish intermediate progress by awaiting the active RolloutContext.set_status() method. Result polling no longer reads progress from ExecutionBackend.rollout_status().
• osmosis eval upload and osmosis eval run --upload include the combined logs.txt, so local evaluation logs appear in the platform’s run Logs tab.
• Local evaluation redacts known provider and platform credentials of at least eight characters found in the process environment from logs, and repeats the redaction before upload hashing.

​

Who needs to act

• Update callers that treated the return value of run_rollout_async() as an asyncio.Task. Keep awaiting the handle for the terminal result, and use its done(), cancel(), and milestone methods instead of Task-only APIs.
• Update custom backends that reported intermediate progress through ExecutionBackend.rollout_status() to await RolloutContext.set_status() instead.

​

What changed

• Rollout completion now uses leased long polling instead of callbacks. The new RolloutClient submits a rollout, renews the server-issued polling lease automatically, retries admission on HTTP 429, polls for the terminal result, and supports explicit cancellation. A request can skip grading with grade=False.
• HttpRolloutDriver and the callback models are removed. Each request supplies a unique rollout_id, plus an explicit llm_api_key when the chat endpoint requires authentication.
• Custom backends return an ExecutionOutcome from execute(request) instead of invoking result callbacks. Local LLM bridge imports move from the removed osmosis_ai.rollout.controller package to osmosis_ai.eval.local.
• Supported Harbor versions expand to >=0.20.0,<0.23, and OpenAI Agents to >=0.18.1,<0.21.
• Polled results preserve completed rewards and failure details through cancellation and serialization errors, and lease expiry or server shutdown allows bounded workflow and sandbox cleanup.
• Admission deadlines cover HTTP requests and retry delays. A lost admission response leaves unobserved work to lease expiry instead of risking cancellation of another rollout with the same ID.
• Local evaluation bounds result polling, waits for cancellation cleanup, and leaves work without a recorded terminal result pending for resume, including on polling 403 and 404 errors.

​

Who needs to act

• Upgrade callers and rollout servers together. The leased long-poll protocol does not interoperate with the 0.3.2 callback protocol, so replace HttpRolloutDriver and callback models with RolloutClient.
• Local evaluation’s protocol fingerprint is now 0.4. A run recorded with the previous protocol cannot resume under this release; use a new run name, or finish the run with the previous SDK.
• When upgrading Harbor, remove top-level trajectory.json inputs from SDK workflow tasks, and ensure the Docker host passes its nftables probe before running restricted-network trials.

​

Breaking Changes

• The public MessageResult, GraderInitRequest, GraderInitResponse, RolloutDriver, and resolve_workspace_directory_from_cwd() APIs are removed. Use OperationResult, HttpRolloutDriver, and resolve_workspace_directory() instead; the two unused grader-init types have no replacement.
• The SDK no longer declares requests as a base dependency or tqdm in the rubric extra. Declare either package directly if your own code imports it.

​

Added

• osmosis eval run reaches Daytona, SkyPilot, and other cloud sandboxes through a managed cloudflared tunnel that starts automatically when the resolved sandbox cannot reach host loopback, or through a tunnel you run yourself with --advertise-url.
• Root --workspace <name> selects a platform workspace by name. Benchmark, dataset, model, secret, and run-inspection commands then work without a local repository, and eval submit and train submit accept an absolute config path when its Git repository matches the selected workspace.
• osmosis eval upload <run-name> resolves a completed run under .osmosis/evals/; explicit run directories still work.
• The eval installation extra is documented, and Python 3.14 is supported.

​

Changed

• Local osmosis eval run --dataset-file ... no longer loads platform credentials unless --upload is requested.
• Newly scaffolded rollouts depend on the stable osmosis-ai[server]>=0.3.0,<0.4 release line.
• Local evaluation output, retry, resume, and upload paths are printed relative to the directory the command was invoked from.

​

Fixed

• Local evaluation validates the model and rollout server before opening a tunnel, keeps slow non-streaming tunnel responses alive, accepts a registered Cloudflare connection when the host cannot probe the tunnel URL, and fails fast on an unreachable loopback endpoint instead of hanging.
• Local evaluation warns when the CLI and the rollout environment use different osmosis-ai versions.
• The harbor extra installs Harbor’s Daytona dependencies, so Daytona environments work without separate dependency setup.
• --secrets-file values override the process environment only for the local evaluation run and are restored on every exit path, and orphan cleanup rejects symlinked run directories.

​

Upgrade

​

Breaking Changes

• RolloutDriver.run now takes a single RolloutRunRequest argument. Update custom drivers and their callers to pass the request object.
• ExecutionBackend.max_concurrency and the import-time osmosis_ai.platform.auth.PLATFORM_URL are removed. Read concurrency capacity from the rollout server’s /health response, and the active platform URL from get_platform_url().
• Unused server-owned fields are removed from the public record types. Stop reading UploadInfo.s3_key and .upload_id, DatasetFile.df_stats and .organization_id, TrainingRunMetrics.training_run_id, EvalRunMetrics.eval_run_id, RolloutInfo.last_synced_at, and TrainingRunCheckpoints.training_run_id.

​

Added

• osmosis eval run executes a crash-safe local evaluation from the same TOML a managed run uses, through the new eval installation extra. It supports dataset slicing, resumable output, uv-managed rollout environments, LocalBackend and Harbor’s Docker environment, readable generated run names, OpenAI Responses routing, bounded admission, and orphan rollout-server cleanup.
• osmosis eval upload <run-dir> and eval run --upload import a completed local run into the platform. Both are server-authoritative and idempotent, and neither launches a managed evaluation.

​

Changed

• The CLI machine contract is standardized. --json and --plain never prompt, JSON errors use stable {code, message, details} envelopes on stderr, machine-readable warnings use JSON Lines, and non-finite values can no longer produce invalid JSON.
• Logins are platform-scoped and persist in the operating-system keyring across directories and environments. They survive HTTP 401 responses, and a non-production OSMOSIS_TOKEN is validated against OSMOSIS_TOKEN_PLATFORM_URL before any network access.

​

Fixed

• Mini SWE-agent benchmark credential validation matches the Platform, while the Cursor CLI harness-key requirement is preserved.
• osmosis dev server up prints the one-time API key the Platform returns, so the provisioned server can be used immediately.

​

Upgrade

​

Breaking Changes

• Every rollout now produces exactly one RolloutSample and reward through rollout-scoped URLs. Update custom contexts, graders, integrations, callbacks, and backend adapters to the single-sample contract.
• Runtime integrations use explicit server, strands, openai-agents, harbor, rubric, and parquet extras and feature-specific imports. AgentWorkflow.run() returns one message history, and the former loader, validator, and multi-trajectory helpers are removed.
• HarborBackend is now the container-native implementation previously called HarborBackendV2; the legacy backend, its constructor arguments, OsmosisInstalledAgent, and HarborAgentWorkflowContext are removed.
• Prompt and metadata datasets each use one uniform schema across every row, and benchmark detail output replaces required_secret_names with requires_judge_api_key.

​

Added

• osmosis benchmark now supports catalog discovery, submission, run inspection, logs, cancellation, and output downloads.
• The container-native Harbor backend adds installable workflow bundles, native agents, template and dataset modes, prewarming, diagnostics, artifacts, admission control, status polling, and cancellation.
• Train, evaluation, and benchmark submissions can resolve per-run secrets from a dotenv file, standard input, process environment, or hidden terminal prompt without saving them to the Platform secret store.
• osmosis quickstart guides users through authentication, workspace repository setup, cloning, billing checks, and a ready-to-paste agent prompt; matching onboarding APIs are available on OsmosisClient.

​

Reliability and Security

• Rollout bundles use content-addressed atomic caches, support flat, src/, and PEP 420 layouts, reject unsafe symlinks and cache layouts, and install a compatible uv builder through the Harbor extra.
• Local execution enforces controller deadlines, validates rewards, sanitizes optional callback telemetry, and preserves consistent terminal callback outcomes.
• Harbor preserves samples, diagnostics, and ATIF trajectories while preventing task-source symlinks and rollout credentials from entering archived trial data; secrets-file errors no longer echo secret-bearing input.

​

Upgrade

​

What changed

• osmosis quickstart now signs in, selects a workspace, waits for its repository connection, clones or reuses the workspace directory, checks billing, and produces a ready-to-paste agent prompt for training, evaluation, or benchmarks.
• OsmosisClient adds list_workspaces(), get_quickstart_status(), and complete_quickstart() for workspace-scoped onboarding integrations.
• RolloutSample.reward now rejects NaN, infinity, and non-numeric values at construction and assignment. A grader that assigns one of these values raises pydantic.ValidationError instead of silently reaching the controller as no reward at all. NumPy-like numeric scalars are normalized to float.
• LocalBackend counts time spent waiting on its concurrency limiter against the controller’s agent_timeout_sec, and applies an independent grader_timeout_sec around grading. Workflows that swallow CancelledError or block the event loop past their deadline are now reported as timeouts instead of returning a late success.
• osmosis eval submit, osmosis train submit, and osmosis benchmark submit accept --secrets-file lines written as export NAME=value and NAME="quoted value". The CLI reports a malformed line by source and line number only, so the secret text never reaches CI logs. The CLI also rejects names that do not pass Python’s identifier check up front.
• Rollout bundle builds are now content-addressed by project contents, interpreter ABI, and platform, and each build runs in an isolated directory that must produce exactly one wheel. src/ layouts (as scaffolded by uv init --lib) and PEP 420 namespace packages are supported alongside flat layouts, and the Harbor extra installs its compatible uv builder.
• Harbor rejects symlinks before task materialization and scrubs rollout controller credentials from retained trial files before they can be archived.

​

Who needs to act

• Upgrade to osmosis-ai>=0.3.0rc4 before using osmosis quickstart or relying on these rollout hardening fixes.
• Update any grader that could assign float("nan"), float("inf"), or a non-numeric object to ctx.set_reward. Osmosis now rejects these values. Return the intended numeric reward, or leave the reward unset to mean “not graded”.
• Update --secrets-file sources that relied on the previous parser accepting quotes as part of a value or an export prefix as part of a name. The old behavior silently submitted the wrong string; the new parser resolves both forms to the intended value.

​

What changed

• Prompt datasets require user_prompt plus ground_truth or its label alias; system_prompt remains optional.
• Metadata datasets require a non-empty metadata object on every row. Validation now checks every JSONL and CSV row and every Parquet metadata value.
• osmosis train submit, osmosis eval submit, and osmosis benchmark submit accept --secrets-file; values can also come from the process environment or an interactive prompt.
• Benchmark details replace required_secret_names with requires_judge_api_key.

​

Who needs to act

​

What changed

• Install optional features with the server, strands, openai-agents, harbor, rubric, or parquet extras; full installs all of them.
• Import server, Harbor, Strands, OpenAI Agents, and rubric features from their public submodules instead of the rollout or package root.
• AgentWorkflow.run() returns one message history as AgentWorkflowOutput, a bare message list, or None for the active sample-source fallback. Output metrics must be finite.
• The former HarborBackendV2 is now HarborBackend; the legacy backend and HarborAgentWorkflowContext were removed.
• The CLI adds benchmark catalog, submission, status, logs, cancellation, and output-download workflows.

​

Who needs to act

​

What changed

• GraderContext.samples is now GraderContext.sample, and set_sample_reward(sample_id, reward) is now set_reward(reward).
• Custom integrations register one source with set_sample_source() and read it with get_sample().
• RolloutSample.id and MultiTurnMode were removed. The rollout URL now supplies execution identity.
• Model and callback requests use rollout-scoped URLs. Integrations no longer attach per-call sample or rollout routing headers.
• Backends exchange sample.json and a single-value reward.json ({"reward": <float>}).
• Strands and OpenAI Agents integrations enforce one registered agent or session per workflow execution.

​

Who needs to act