Summary
TokenFirewall handles OpenAI costs only. Adding Anthropic and Gemini adapters makes it a universal cost layer for the three dominant model provider families.
Problem Statement
Teams using multiple LLM providers must maintain separate cost-tracking solutions per provider. A single TokenFirewall instance should be able to enforce budgets across all providers in use.
Proposed Solution
Define a ProviderAdapter interface:
interface ProviderAdapter {
name: string;
countTokens(text: string, model: string): Promise<number>;
estimateCost(inputTokens: number, outputTokens: number, model: string): number;
interceptRequest(req: Request): LLMRequestMetadata;
interceptResponse(res: Response): LLMResponseMetadata;
}
Anthropic Adapter
- Use
@anthropic-ai/sdk client.beta.messages.countTokens() for pre-request counting.
- Parse
usage.input_tokens and usage.output_tokens from the response body.
- Pricing table for Anthropic model tiers (opus, sonnet, haiku), configurable via env vars.
Gemini Adapter
- Use
@google/generative-ai model.countTokens() for estimation.
- Parse
usageMetadata.promptTokenCount and candidatesTokenCount from responses.
- Support Gemini 1.5 Pro and Flash tiers.
Acceptance Criteria
Summary
TokenFirewall handles OpenAI costs only. Adding Anthropic and Gemini adapters makes it a universal cost layer for the three dominant model provider families.
Problem Statement
Teams using multiple LLM providers must maintain separate cost-tracking solutions per provider. A single TokenFirewall instance should be able to enforce budgets across all providers in use.
Proposed Solution
Define a
ProviderAdapterinterface:Anthropic Adapter
@anthropic-ai/sdkclient.beta.messages.countTokens()for pre-request counting.usage.input_tokensandusage.output_tokensfrom the response body.Gemini Adapter
@google/generative-aimodel.countTokens()for estimation.usageMetadata.promptTokenCountandcandidatesTokenCountfrom responses.Acceptance Criteria
provider: 'anthropic'routes token counting to the Anthropic adapter.provider: 'gemini'routes to the Gemini adapter.