Osmosis AI Python SDK
The Osmosis AI Python SDK provides tools for evaluating LLM outputs using natural language rubrics and reward functions. It supports both local deterministic evaluation and remote LLM-based semantic evaluation across multiple providers.What is Osmosis AI?
Osmosis AI helps you:- Evaluate LLM outputs with natural language rubrics
- Create reward functions for reinforcement learning and scoring
- Compare providers across OpenAI, Anthropic, Gemini, xAI, and more
- Batch process evaluations with a built-in CLI tool
Core Concepts
Osmosis AI provides two evaluation approaches: 1. Local Reward Functions - Fast, deterministic scoring (e.g., exact match, regex) 2. Remote Rubric Evaluation - LLM-powered semantic judgment with natural language criteria See the Quick Start for code examples and hands-on tutorials.CLI Tool
Batch evaluate datasets from the command line:Key Features
LLM-Based Rubrics
Natural language evaluation criteria with semantic understanding
Local Reward Functions
Fast, deterministic functions for exact match and simple checks
Multi-Provider Support
OpenAI, Anthropic, Gemini, xAI, OpenRouter, and Cerebras
CLI Tool
Batch evaluations with statistics and result tracking
Use Cases
- Quality Assurance - Evaluate LLM responses before serving to users
- Model Comparison - Compare outputs across models and providers
- Reinforcement Learning - Create reward functions for training
- A/B Testing - Measure impact of prompt variations