Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,19 @@ Current endpoints:
- `GET /test/token-pilot/budget`: exercises budget enabled/limit behavior when budget beans are present.
- `GET /actuator/prometheus`: validates actuator/prometheus exposure.

Demo profile endpoints (run with `--spring.profiles.active=demo`):

- `GET /test/token-pilot/demo`: describes the no-key, in-memory provider and scenario links.
- `GET /test/token-pilot/demo/run`: runs context fit/block, atomic reservation concurrency,
idempotency, release, and success/failure/unknown reconciliation scenarios.
- `GET /test/token-pilot/demo/{context-fit|context-block|budget-concurrency|idempotency|release|reconciliation-success|reconciliation-failure|reconciliation-unknown}`:
runs one deterministic scenario.

The demo profile uses `DemoChatModel` only; it makes no external provider calls. The
sample app's `DEMO_RUNBOOK.md` documents the Gradle, Prometheus/Grafana, and scenario
verification flow. The provisioned dashboard uses Token Pilot-owned `tokenpilot_*`
meters and deliberately does not use legacy `ai_token_*` meters.

Test-only E2E endpoint:

- `GET /test/token-pilot/chat`: exercises the Spring AI `ChatClient` advisor path with a fake/mock provider or documented real provider setup.
Expand Down Expand Up @@ -463,6 +476,10 @@ Stage and deploy a Central release:
- Added root publication aggregation and a release-candidate verification gate
that stages every public module and checks the complete artifact/signature
set for the selected version.
- Added a `demo` Spring profile to the sample app with a deterministic in-memory
ChatModel, public admission/reservation/idempotency/reconciliation scenarios,
integration coverage, a Token Pilot Prometheus/Grafana dashboard, and a
deterministic verification runbook.
- Added isolated external Core/Starter consumer verification with fresh cache,
dependency evidence, negative artifact/version scenarios, and a protected
0.1.0 release-candidate workflow.
Expand Down
46 changes: 40 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# TokenPilot

> One Java gateway to control LLM providers, routing, fallback, tokens, and costs.
> Java LLM control and accounting with optional provider adapters.

TokenPilot helps Java applications control an LLM call before it is sent and
reconcile its cost after it completes. It brings token estimation, context
Expand Down Expand Up @@ -57,6 +57,41 @@ Spring AI `ChatClient` calls, TokenPilot now performs conservative preflight,
claims one provider dispatch, and reconciles provider-reported actual usage
against the reservation-time pricing snapshot.

## Documentation

- [10-minute quickstart](docs/QUICKSTART.md) — choose Core or Starter and run the first verification path.
- [Configuration reference](docs/CONFIGURATION.md) — properties, defaults, conditions, and failure modes.
- [Metrics reference](docs/METRICS.md) — Token Pilot-owned meters and legacy compatibility policy.
- [Sample app runbook](docs/SAMPLE_RUNBOOK.md) — local app, Prometheus, Grafana, and troubleshooting commands.
- [Deterministic demo runbook](token-pilot-sample-app/DEMO_RUNBOOK.md) — eight admission, reservation, idempotency, release, and reconciliation scenarios.
- [Release procedure](docs/RELEASE.md) — staging, signing, external consumer, and Central Portal gates.
- [30-day MVP cutline](docs/30_DAY_MVP_REPORT.md) and [post-MVP evolution plan](docs/EVOLUTION_PLAN.md).

## Sample demo

The sample app exposes a deterministic `demo` profile for checking the control
and accounting lifecycle from HTTP. It covers context admission, atomic budget
reservation, idempotency, release, successful reconciliation, and pending
reconciliation results.

```bash
./gradlew --no-daemon :token-pilot-sample-app:bootRun \
--args='--spring.profiles.active=demo'

curl -s http://localhost:8080/test/token-pilot/demo/run | jq
```

The demo returns eight `PASS` scenario results and a snapshot of the current
Token Pilot-owned metrics. Prometheus and Grafana can be started with:

```bash
docker compose -f token-pilot-sample-app/docker-compose.yml up --build -d
```

See [QUICKSTART.md](docs/QUICKSTART.md) for the ten-minute path and
[SAMPLE_RUNBOOK.md](docs/SAMPLE_RUNBOOK.md) for endpoint and troubleshooting
details.

## Micrometer metrics

When a `MeterRegistry` is available, the Spring Boot starter publishes
Expand Down Expand Up @@ -183,11 +218,10 @@ Boot, Spring AI, Micrometer, or Reactor dependencies. Spring Boot 3, Spring AI
1.x, other Spring Boot/Spring AI patch combinations, and older Java runtimes
are not part of the 0.1.0 support guarantee.

The verified Spring AI path is the synchronous `ChatClient` call lifecycle with
a fake provider, including preflight blocking, atomic reservation, dispatch,
and estimate/actual reconciliation. Real-provider compatibility, chunk
accounting, streaming cancellation, and partial-usage reconciliation are not
included in this compatibility claim.
The Spring AI path includes the synchronous `ChatClient` call lifecycle with
preflight blocking, atomic reservation, dispatch, and estimate/actual
reconciliation. Chunk accounting, streaming cancellation, and partial-usage
reconciliation remain outside the current lifecycle.

If a provider returns a model different from the request pricing snapshot,
TokenPilot keeps the estimate as `PRICING_RECONCILIATION_REQUIRED` instead of
Expand Down
46 changes: 46 additions & 0 deletions docs/30_DAY_MVP_REPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# 30-day MVP report

This document is the active MVP cutline for Token Pilot 0.1.0. It describes
verified repository behavior, not a promise that every artifact is already
published.

## Current release-candidate scope

| Area | Status |
| --- | --- |
| Usage normalization and token breakdown invariants | Implemented and unit tested |
| Exact monetary arithmetic and currency validation | Implemented with `BigDecimal` and explicit currency |
| Missing-pricing policy and immutable pricing snapshot | Implemented |
| Versioned model catalog and context admission | Implemented |
| UTF-8 byte heuristic estimator | Implemented; `TEXT_ONLY`, heuristic, not exact BPE |
| Atomic in-memory budget reservation | Implemented |
| Estimate/actual reconciliation | Implemented, including pending liability and duplicate-safe callbacks |
| Threshold notification events | Implemented as process-local best-effort delivery |
| Token Pilot-owned Micrometer metrics | Implemented with bounded default tags |
| Spring AI 2.0 non-streaming lifecycle | Implemented and fake-provider E2E tested |
| Sample app deterministic demo profile | Implemented with eight HTTP scenarios and Prometheus/Grafana verification |
| Core-only and Starter publication paths | Build and generated consumer gates present; final staging evidence required |
| Durable stores/outbox/replay | Post-MVP |
| Advanced streaming reconciliation | Post-MVP |

## Release exit criteria

- `./gradlew test` passes.
- Publication metadata, MIT license metadata, Java 25 bytecode, and dependency boundaries pass verification.
- `token-pilot-core` resolves from an external consumer without Spring, Spring AI, Micrometer, or Reactor runtime dependencies.
- `token-pilot-starter` resolves with a provider selected by the application and does not bundle a provider implementation.
- Core and Starter use the same release version.
- The default Token Pilot metrics do not expose raw user, tenant, request, reservation, or model identifiers.
- The sample app can expose health, smoke, deterministic ledger, and Prometheus checks without an API key.
- Release documentation does not describe staging artifacts as a Maven Central release.

## Explicit cutline

The following are deliberately outside the 0.1.0 MVP:

- Exact provider BPE/tokenizer assets.
- Full media, tools, schemas, and structured-output admission.
- Streaming chunk accounting and cancellation reconciliation.
- Durable reservation/accounting/notification delivery.
- Provider routing, retries, fallback, circuit breakers, and gateway HTTP runtime.
- Multi-tenant administration and durable operational stores.
180 changes: 180 additions & 0 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
# Token Pilot configuration reference

All Token Pilot Spring Boot properties use the `token-pilot` prefix. The
defaults below are supplied by `TokenPilotProperties` and the owner-specific
auto-configurations.

## Root and pricing

| Property | Default | Meaning |
| --- | --- | --- |
| `token-pilot.enabled` | `true` | Enables the Token Pilot auto-configuration graph. |
| `token-pilot.pricing.plans` | `[]` | Pricing plans bound into the in-memory pricing registry. |
| `token-pilot.pricing.plans[].model-id` | required | Model ID or exact alias resolved by the pricing registry. |
| `token-pilot.pricing.plans[].currency` | required | ISO-4217 currency code used by the plan. |
| `token-pilot.pricing.plans[].rates.PROMPT` | plan-defined | Rate for ordinary input tokens. |
| `token-pilot.pricing.plans[].rates.COMPLETION` | plan-defined | Rate for ordinary output tokens. |
| `token-pilot.pricing.plans[].rates.REASONING` | optional | Rate for reasoning output tokens. |
| `token-pilot.pricing.plans[].rates.CACHE_READ_PROMPT` | optional | Rate for cached input tokens read from cache. |
| `token-pilot.pricing.plans[].rates.CACHE_CREATION_PROMPT` | optional | Rate for input tokens written to cache. |

Rates use `BigDecimal` internally. A missing plan or rate is not silently
converted into a priced zero. The configured missing-pricing policy determines
whether the provider boundary fails closed or records an unpriced result.

```yaml
token-pilot:
pricing:
plans:
- model-id: gpt-4o-mini
currency: USD
rates:
PROMPT: 0.00015
COMPLETION: 0.00060
```

## Spring AI adapter

| Property | Default | Meaning |
| --- | --- | --- |
| `token-pilot.spring-ai.default-model-id` | unset | Fallback model when the request/provider does not expose a model. |
| `token-pilot.spring-ai.default-reserved-output-tokens` | unset | Fallback output reservation when request options and a resolver provide none. |
| `token-pilot.spring-ai.framing-headroom-tokens` | `0` | Additional conservative headroom for known request framing or advisor additions. |

Resolution order for reserved output is:

```text
ChatOptions.maxTokens → ReservedOutputTokensResolver → configured default
```

Non-positive resolver values are rejected. The current request admission scope
is `TEXT_ONLY`; media, tool messages or schemas, structured-output
augmentation, and enforcement-enabled streaming are rejected before
reservation when they cannot be represented by the supported contract.

## Metrics

| Property | Default | Meaning |
| --- | --- | --- |
| `token-pilot.metrics.enabled` | `true` | Enables Token Pilot-owned meters when a `MeterRegistry` exists. |
| `token-pilot.metrics.tag-whitelist` | `[]` | User tag keys allowed only on the legacy compatibility publisher. |
| `token-pilot.metrics.legacy-ai-token-metrics-enabled` | `false` | Opts into the legacy `ai.token.*` meters. |

The default Token Pilot-owned meters use bounded domain values and do not add
raw model, tenant, user, request, reservation, or idempotency identifiers.
`tag-whitelist` limits keys, not the number of distinct values supplied by an
application.

```yaml
token-pilot:
metrics:
enabled: true
tag-whitelist: []
legacy-ai-token-metrics-enabled: false
```

## Budget

| Property | Default | Meaning |
| --- | --- | --- |
| `token-pilot.budget.enabled` | `false` | Enables the in-memory atomic reservation and accounting store. |
| `token-pilot.budget.monthly-limit` | `10.00` | Monthly limit for each resolved budget key. |
| `token-pilot.budget.policy-id` | `default-monthly` | Policy identity included in the budget key. |
| `token-pilot.budget.target-type` | `tenant` | Target namespace included in the budget key. |
| `token-pilot.budget.target-tag-key` | `tenant_id` | Request tag used to resolve the target ID. |
| `token-pilot.budget.fallback-target-id` | unset | Explicit fallback when the target tag is absent. Without it, resolution fails closed. |
| `token-pilot.budget.currency` | `USD` | Currency of the monthly limit. |
| `token-pilot.budget.zone-id` | `UTC` | IANA zone used to define monthly window boundaries. |

When enforcement is enabled, the safe preflight upper bound is reserved before
provider dispatch. Actual usage later moves liability into committed cost, or
into a pending reconciliation state when usage is unavailable or needs a
separate pricing snapshot.

```yaml
token-pilot:
budget:
enabled: true
policy-id: default-monthly
target-type: tenant
target-tag-key: tenant_id
fallback-target-id: shared
monthly-limit: 10.00
currency: USD
zone-id: Asia/Seoul
```

Budget state is process-local and in-memory. It does not provide a durable
store, restart replay, or multi-instance atomicity.

## Notification

| Property | Default | Meaning |
| --- | --- | --- |
| `token-pilot.notification.enabled` | `false` | Enables threshold event deduplication and notification lifecycle events. |

Notification auto-configuration requires a `BudgetNotificationHandler` bean
and an enabled budget accounting graph:

```java
@Component
final class BudgetAlertHandler implements BudgetNotificationHandler {
@Override
public void handle(BudgetNotificationEvent event) {
// Application-owned email, Slack, Webhook, or audit integration.
}
}
```

Handler failures are isolated from the provider response and accounting
transition. Delivery is process-local, best-effort, and at-most-once.

## Sample `demo` profile

The sample app's `demo` profile supplies a deterministic in-memory `ChatModel`,
enables budget enforcement, and exposes the eight scenario endpoints described
in [QUICKSTART.md](QUICKSTART.md). Its effective configuration is:

```yaml
token-pilot:
enabled: true
pricing:
plans:
- model-id: gpt-4o-2024-08-06
currency: USD
rates:
PROMPT: 0.00015
COMPLETION: 0.00060
spring-ai:
default-model-id: gpt-4o-2024-08-06
default-reserved-output-tokens: 16
framing-headroom-tokens: 0
budget:
enabled: true
policy-id: demo-monthly
target-type: tenant
target-tag-key: tenant_id
fallback-target-id: demo-shared
monthly-limit: 0.001
currency: USD
zone-id: UTC
metrics:
enabled: true
legacy-ai-token-metrics-enabled: false
tag-whitelist: []
```

Start it with:

```bash
./gradlew --no-daemon :token-pilot-sample-app:bootRun \
--args='--spring.profiles.active=demo'
```

## Auto-configuration rules

- User-defined beans win over defaults through `@ConditionalOnMissingBean`.
- Optional integrations require their classpath and feature conditions.
- Budget and notification beans are absent by default.
- A custom budget store used for enforcement must also expose the accounting capability; startup fails rather than silently falling back to ledger-only behavior.
- The starter does not select an LLM provider or require an API key by itself.
36 changes: 36 additions & 0 deletions docs/EVOLUTION_PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Evolution plan

Token Pilot is starting as a framework-independent Java LLM control and
accounting core with optional adapters. The long-term direction is a Java-native
embeddable gateway SDK and, later, an optional standalone runtime.

## Post-MVP workstreams

### 1. Exact token intelligence

- Add exact byte-level BPE implementations and versioned encoding assets.
- Expand Unicode and provider corpus coverage.
- Benchmark estimators with JMH and make tokenizer compatibility explicit.
- Preserve the existing safe-upper-bound contract while exact implementations mature.

### 2. Provider-independent gateway slice

- Define provider-neutral request, response, usage, and error contracts.
- Add an optional OpenAI-compatible HTTP gateway surface.
- Keep the core accounting and budget interfaces reusable by direct SDK consumers and gateway callers.

### 3. Reliable routing and execution

- Add retries, fallback, circuit breakers, and provider attempt identity.
- Reconcile cost and usage at attempt level.
- Define streaming cancellation and partial-usage semantics before enabling enforcement for streaming calls.

### 4. Durable multi-tenant operations

- Add durable reservation, accounting, notification, and audit stores.
- Support multi-instance atomicity and restart replay.
- Add tenant policy administration and security hardening.
- Add OpenTelemetry integration where it complements, rather than duplicates, Spring AI telemetry.

These workstreams must not be presented as implemented features until their
contracts, tests, and external-consumer evidence land.
Loading
Loading