0.3.md

Migrating to TinyInference 0.3

TinyInference 0.3 makes model-boundary metadata first class. This release is a pre-1.0 minor bump because the public stream and request/response shapes change.

Streams

ModelStream is no longer a type alias. This is a source-breaking public API change: custom ChatModel implementations that returned Ok(Box::pin(items)) must wrap the stream explicitly:

let stream = ModelStream::new(Box::pin(items));

Use with_correlation and with_resolved_route to set metadata for all stream items. with_correlation also stamps a terminal Completed response that did not already carry the request correlation. Detached producers should use abort_on_drop(AbortOnDrop::from_join_handle(&handle)).

Calls and routes

Set ModelRequest::with_correlation before dispatch. Built-in providers carry it to unary responses, stream metadata, and terminal responses. model is a provider model id or alias; use with_requested_route for a host route. Generic fallback observation compares requested and resolved route names only.

Responses and embeddings

Initialize ModelResponse::correlation and resolved_route in custom struct literals, or use the response builders. Usage now has typed charged_amount and context_window_tokens fields.

Call EmbeddingModel::embed_request(EmbeddingRequest::new(inputs)) when the caller needs validated vector count/dimensions, provider usage, or cancellation. Cancelling its EmbeddingCancellation drops the active provider future.

Decision crate

The Cargo package is now named tinyinference-decisions. Update dependency declarations and cargo -p commands that used tinyinference_decisions. Rust imports remain tinyinference_decisions. Levanto Sage is available in tinyinference_decisions::sage; existing Jev imports remain at the crate root.