chat-rs

A multi-provider LLM framework for Rust. Build type-safe chat clients with tool calling, structured output, streaming, and embeddings — swap providers with a single line change.

Features

• Multi-provider — Gemini, Claude, OpenAI, and Router today, more coming (see Roadmap)
• Router — route requests across multiple providers with fallback and custom strategies (keyword, embedding, capability-based)
• Type-safe builder — compile-time enforcement of valid configurations via type-state pattern
• Tool calling — define tools with #[tool], the framework handles the call loop automatically
• Structured output — deserialize model responses directly into your Rust types via schemars
• Streaming — real-time token-by-token output with tool call support
• Embeddings — generate vector embeddings through the same unified API
• Retry & callbacks — configurable retry strategies with before/after hooks
• Native tools — provider-specific features like Google Search, code execution, web search

Quick Start

Add to your Cargo.toml:

[dependencies]
chat-rs = { version = "0.0.10", features = ["openai"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

use chat_rs::{ChatBuilder, openai::OpenAIBuilder, types::messages};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let client = OpenAIBuilder::new().with_model("gpt-4o-mini").build();
    let mut chat = ChatBuilder::new().with_model(client).build();

    let mut messages = messages::from_user(vec!["Hey there!"]);
    let res = chat.complete(&mut messages).await?;
    println!("{:?}", res.content);

    Ok(())
}

Set your API key via environment variable (OPENAI_API_KEY, GEMINI_API_KEY, or CLAUDE_API_KEY), or pass it explicitly with .with_api_key().

Providers

Enable providers via feature flags:

# Pick one or more
chat-rs = { version = "0.0.10", features = ["gemini"] }
chat-rs = { version = "0.0.10", features = ["claude"] }
chat-rs = { version = "0.0.10", features = ["openai"] }
chat-rs = { version = "0.0.10", features = ["router", "gemini", "claude"] }
chat-rs = { version = "0.0.10", features = ["gemini", "claude", "openai", "stream"] }

Provider	Feature	API Key Env Var	Builder
Google Gemini	gemini	GEMINI_API_KEY	GeminiBuilder
Anthropic Claude	claude	CLAUDE_API_KEY	ClaudeBuilder
OpenAI	openai	OPENAI_API_KEY	OpenAIBuilder
Router	router	—	RouterBuilder

Swapping providers is a one-line change — replace the builder, everything else stays the same:

// Gemini
let client = GeminiBuilder::new()
    .with_model("gemini-2.5-flash".to_string())
    .build();

// Claude
let client = ClaudeBuilder::new()
    .with_model("claude-sonnet-4-20250514".to_string())
    .build();

// OpenAI
let client = OpenAIBuilder::new()
    .with_model("gpt-4o")
    .build();

// Same from here on
let mut chat = ChatBuilder::new().with_model(client).build();

Tool Calling

Define tools with the #[tool] macro from tools-rs and register them with collect_tools(). The framework automatically loops through tool calls until the model is done.

use chat_rs::{ChatBuilder, gemini::GeminiBuilder, types::messages::content};
use tools_rs::{collect_tools, tool};

#[tool]
/// Looks up the current weather for a given city.
async fn get_weather(city: String) -> String {
    format!("The weather in {} is sunny, 22°C", city)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = GeminiBuilder::new()
        .with_model("gemini-2.5-flash".to_string())
        .build();

    let tools = collect_tools();

    let mut chat = ChatBuilder::new()
        .with_tools(tools)
        .with_model(client)
        .with_max_steps(5)
        .build();

    let mut messages = messages::Messages::default();
    messages.push(content::from_user(vec!["What's the weather in Tokyo?"]));

    let response = chat.complete(&mut messages).await.map_err(|e| e.err)?;
    println!("{:?}", response.content);

    Ok(())
}

Structured Output

Deserialize model responses directly into typed Rust structs. Your type must derive JsonSchema and Deserialize.

use schemars::JsonSchema;
use serde::Deserialize;

#[derive(JsonSchema, Deserialize, Clone, Debug)]
struct User {
    pub name: String,
    pub likes: Vec<String>,
}

let mut chat = ChatBuilder::new()
    .with_structured_output::<User>()
    .with_model(client)
    .build();

let response = chat.complete(&mut messages).await?;
println!("Name: {}, Likes: {:?}", response.content.name, response.content.likes);

Streaming

Enable the stream feature flag:

chat-rs = { version = "0.0.10", features = ["gemini", "stream"] }

use chat_rs::StreamEvent;
use futures::StreamExt;

let mut chat = ChatBuilder::new()
    .with_model(client)
    .build();

let mut stream = chat.stream(&mut messages).await?;

while let Some(chunk) = stream.next().await {
    match chunk? {
        StreamEvent::TextChunk(text) => print!("{}", text),
        StreamEvent::ReasoningChunk(thought) => print!("[thinking] {}", thought),
        StreamEvent::ToolCall(fc) => println!("[calling {}]", fc.name),
        StreamEvent::ToolResult(fr) => println!("[tool returned]"),
        StreamEvent::Done(_) => break,
    }
}

Embeddings

let client = GeminiBuilder::new()
    .with_model("gemini-embedding-001".to_string())
    .with_embeddings(Some(768))
    .build();

let mut chat = ChatBuilder::new()
    .with_model(client)
    .with_embeddings()
    .build();

let response = chat.embed(&mut messages).await?;
println!("{:?}", response.embeddings);

Native Tools

Provider-specific capabilities beyond standard tool calling:

// Gemini: Google Search, Code Execution, Google Maps
let client = GeminiBuilder::new()
    .with_model("gemini-2.5-flash".to_string())
    .with_google_search()
    .with_code_execution()
    .build();

// OpenAI: Web Search
let client = OpenAIBuilder::new()
    .with_model("gpt-4o")
    .with_web_search(Some(SearchContextSizeEnum::High), None)
    .build();

OpenAI-Compatible Endpoints

Use local or proxy servers that implement the OpenAI Responses API:

let client = OpenAIBuilder::new()
    .with_model("llama3")
    .with_custom_url("http://localhost:11434/v1".to_string())
    .with_api_key("ollama".to_string())
    .build();

Note: The custom endpoint must support the Responses API format (POST /responses), not the Chat Completions API.

Router

Route requests across multiple providers with automatic fallback on retryable errors. Add a custom RoutingStrategy to control provider selection based on keywords, embeddings, capabilities, or any logic you need.

use chat_rs::{
    ChatBuilder,
    router::RouterBuilder,
    gemini::GeminiBuilder,
    claude::ClaudeBuilder,
    types::messages,
};

let gemini = GeminiBuilder::new()
    .with_model("gemini-2.5-flash".to_string())
    .build();

let claude = ClaudeBuilder::new()
    .with_model("claude-sonnet-4-20250514".to_string())
    .build();

let router = RouterBuilder::new()
    .add_provider(gemini)
    .add_provider(claude)
    // .with_strategy(my_strategy)  // optional custom routing
    .build();

let mut chat = ChatBuilder::new().with_model(router).build();

let mut msgs = messages::from_user(vec!["Hello!"]);
let res = chat.complete(&mut msgs).await?;

Without a custom strategy, the router tries providers in order and falls back on retryable errors (rate limits, network issues). Non-retryable errors are returned immediately.

Note: The router currently supports completions only — streaming and embeddings are not yet available.

Architecture

chat-rs (root)              ← Re-exports + feature flags
├── core/                   ← Traits, types, Chat engine, builder
├── providers/
│   ├── gemini/             ← Google Gemini provider
│   ├── claude/             ← Anthropic Claude provider
│   ├── openai/             ← OpenAI Responses API provider
│   └── router/             ← Multi-provider router
└── examples/
    ├── gemini/             ← Gemini examples
    ├── claude/             ← Claude examples
    ├── openai/             ← OpenAI examples
    └── router/             ← Router strategy examples

See core/AGENTS.md and providers/AGENTS.md for detailed architecture documentation.

Examples

Run examples with the appropriate feature flags:

# Gemini
cargo run --example gemini-tools --features gemini
cargo run --example gemini-structured --features gemini
cargo run --example gemini-stream --features gemini,stream
cargo run --example gemini-embeddings --features gemini
cargo run --example gemini-code-execution --features gemini
cargo run --example gemini-google-maps --features gemini
cargo run --example gemini-image-understanding --features gemini

# Claude
cargo run --example claude-completion --features claude
cargo run --example claude-stream --features claude,stream

# OpenAI
cargo run --example openai-completion --features openai
cargo run --example openai-stream --features openai,stream
cargo run --example openai-structured --features openai
cargo run --example openai-embeddings --features openai

# Router
cargo run --example router-keyword --features router,gemini,claude
cargo run --example router-embeddings --features router,gemini,claude
cargo run --example router-capability --features router,gemini,claude

# Retry strategies
cargo run --example retry --features gemini

Minimum Supported Rust Version

Rust 1.94 or later (edition 2024).

License

MIT