Multi-Provider LLM Architecture

[sbharatjoshi]

🗣️ Discussion: Multi-Provider LLM Architecture

Architectural discussion about Spector's LLM provider abstraction layer — supporting Ollama, OpenAI, Anthropic, and Gemini through a unified SPI.

---

Architecture

┌────────────────────────────────────────────┐
│           LlmBridge (SPI)                  │
│  chat() | chatStream() | embed()          │
└─────────┬─────────┬──────────┬────────────┘
          │         │          │
  ┌───────▼──┐ ┌────▼─────┐ ┌─▼──────────┐
  │ Ollama   │ │ OpenAI   │ │ Anthropic  │ ...
  │ Provider │ │ Provider │ │ Provider   │
  └──────────┘ └──────────┘ └────────────┘
       ↑            ↑            ↑
   ServiceLoader (ADR-003)

Provider Comparison

Feature	Ollama	OpenAI	Anthropic	Gemini
Auth	None	Bearer token	x-api-key	Query param
Streaming	NDJSON	SSE	SSE (custom)	SSE
Tool calling	JSON	tool_calls[]	tool_use blocks	functionCall
Embeddings	/api/embed	/v1/embeddings	❌ (use Voyage)	textembedding
Local	✅	❌	❌	❌
Cost	Free	Pay-per-token	Pay-per-token	Free tier

Design Decisions

1. ServiceLoader SPI (ADR-003) — Providers discovered via META-INF/services, zero reflection
2. No SDK dependencies — We use java.net.http.HttpClient directly for minimal footprint
3. Resilience4j wrapping — Circuit breaker, retry, and timeout on every call
4. Unified streaming — All providers normalize to AgentChatListener.onContent(token)

Questions for Discussion

• Should we support LangChain4j as an alternative provider backend?
• How should we handle provider-specific features that don't map to the unified interface?
• Should embedding providers be separate from chat providers?
• How should model selection work — per-agent, per-request, or global default?
• Should we add a cost tracking layer to monitor token usage across providers?

Related Issues

#95 LLM Provider SPI, #96 Ollama, #125 OpenAI, #126 Anthropic, #127 Gemini, #128 Circuit Breaker

---

Share your thoughts!