diff --git a/.planning/v1.11.1-build/qa-LEDGER.md b/.planning/v1.11.1-build/qa-LEDGER.md new file mode 100644 index 000000000..e23b6b9f9 --- /dev/null +++ b/.planning/v1.11.1-build/qa-LEDGER.md @@ -0,0 +1,32 @@ +# v1.11.1 W10 QA LEDGER + +## Reviewers (6): qa-code-review, qa-security, qa-senior-dev, qa-product-lead, qa-simplifier, qa-i18n-deadcode +Verdicts so far: security PASS (0C/0H/0M), code-review SHIP (0C/0H/1M), product-lead SHIP (0C/0H/2M), senior-dev ship-able (0C/0H/1M). simplifier + i18n pending. + +## RECONCILED (fixed this session) +- [FIXED commit pending] M (code-review M-1 / security Info-1 / product-lead M2 / senior-dev L2): conversation-summary.ts fold-loop decrypt not fault-isolated → wrapped in try/catch (flatMap skip). 4 reviewers flagged same spot. +- [FIXED] M (senior-dev): live/rollup tiebreak divergence for ranked types with only off-ladder sources → unified JS collapse + rollup all-time SQL on `source` ASC (matches canonicalMeasurementsFrom + route CTE). Unit test updated. +- [FIXED] L (security L-1): stale `/^[A-Z_]+$/` doc comment in source-rank-sql.ts → `/^[A-Z0-9_]+$/`. + +## MUST DO IN RELEASE COMMIT (product-lead M1) +- bump package.json 1.11.0→1.11.1 + `pnpm openapi:generate` (else openapi:check reds — v1.6.0/v1.8.5 trap). + +## DEFERRED (Low — backlog, documented) +- code-review L-2 / fast-path redundant loadUserSourcePriority lookups (perf nit; thread userPriorityJson param) — defer. +- code-review L-3 / facts soft-cap drifts >50 (injection bounded top-8) — defer. +- code-review L-4 / route rollup read dropped take:cap (window-bounded) — defer. +- senior-dev L1 / 60s snapshot cache vs fact-delete (≤60s staleness) — CHANGELOG note. +- senior-dev L3 / groupBy=day list view sums all sources while chart is source-aware — defer + note (raw-data view vs canonical chart). +- senior-dev L4 / pre-existing delete-then-insert P2002 race widened by per-source range delete (queue retry mitigates) — defer. +- security Low-2/Info — central redaction covers worker err.message; background-only. +- CHANGELOG known-limitations: latest-tile canonical-source shift (intended R1) + 60s fact-deletion staleness. + +## PENDING: read qa-simplifier.md + qa-i18n-deadcode.md when they land; reconcile any new Medium+. + +## LAST 2 REVIEWERS (simplifier + i18n/knip) — reconciled +- [FIXED] HIGH (i18n/knip + simplifier S1): 3 net-new unused exports failing the enforcing knip Dead-code CI gate — SUMMARY_PROMPT_VERSION + SUMMARY_TARGET_CHARS (conversation-summary.ts) + FACTS_PROMPT_VERSION (facts.ts). Fix: deleted the 2 pure version constants (no consumer), and CONSUMED SUMMARY_TARGET_CHARS as a real safety cap on the stored summary length. Post-fix knip: 3 gone, exit clean. typecheck clean, 47 compute tests green. +- [DEFERRED — Low/perf, rationale] simplifier S2 / code-review L-2: the userPriorityJson load-once param is threaded by zero callers → fast-paths make 2-4 redundant loadUserSourcePriority lookups/request. Split rating (code-review Low, simplifier Medium); impact = sub-ms indexed PK reads, NOT correctness/security/safety; the param already exists for when it matters. Deferred to backlog to avoid ship-time scope creep across 3 loop sites. +- [DEFERRED — Low] simplifier: ExtractOpts.now dead field (facts.ts) — harmless forward-compat; stale measurement-read.ts docstring ("max count" vs source-name tiebreak) — cosmetic; residual hand-inlined canonicalMeasurementsFrom copy in summaries-slice — equivalent SQL. All Low, backlogged. +- i18n: CLEAN (zero new t() calls, no UI files, prompts server-composed). openapi:check IN SYNC. + +## FINAL QA TALLY: 0 Critical / 0 High open / 0 Medium open. All Medium+ reconciled (decrypt isolation, tiebreak parity, knip unused exports). Lows deferred with rationale. SHIP-CLEARED. diff --git a/CHANGELOG.md b/CHANGELOG.md index e3c3aa54d..95d6fb2b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,24 @@ # Changelog +## [1.11.1] — 2026-06-04 — source-aware vitals + Coach long-term memory + +Closes the three follow-ups left open by v1.11.0: cross-source vital de-duplication, the Coach's conversation memory, and durable personal facts. + +### Added + +- **Coach long-term memory.** The assistant now keeps a private, encrypted rolling summary of long conversations and remembers durable facts you have told it — standing preferences, goals, constraints, and conditions you have stated about yourself. Both are descriptive recall, never a diagnosis, and ride the same Coach setting; turning the Coach off stops all of it. +- **A facts surface to review and clear what the Coach has learned.** List your active facts, forget a single one, or forget everything (`GET` / `DELETE /api/insights/coach/facts`, `DELETE /api/insights/coach/facts/{id}`). Facts are stored encrypted at rest. + +### Changed + +- **Measurement rollups are now source-aware.** When two sources report the same standard vital on the same day (for example a resting heart rate from two devices), charts, summaries, and insights now show the higher-priority source's reading instead of a blend of the two. Cumulative metrics such as steps and energy continue to total per source. Your source-priority settings decide which reading wins. + +### Notes + +- The "latest" value shown for a vital with two competing sources now follows your source priority for the most recent day, so it matches the chart line. For some metrics this may surface a different device's reading than before. +- After clearing a Coach fact it can take up to a minute to disappear from the assistant's working context, in line with the existing snapshot refresh window. +- On first start after upgrading, each account's rollup cache is rebuilt once in the background; charts fall back to a live read until it converges. + ## [1.11.0] — 2026-06-04 — WHOOP, a longitudinal coach, and a clinician-grade record A multi-feature milestone across three fronts: a second connected provider, deeper Insights, and a shareable clinical record. diff --git a/docs/api/openapi.yaml b/docs/api/openapi.yaml index 51e43b146..39ef54640 100644 --- a/docs/api/openapi.yaml +++ b/docs/api/openapi.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: HealthLog API - version: 1.11.0 + version: 1.11.1 description: >- Self-hosted personal-health-tracking PWA — public API surface for the iOS native client and external ingest. @@ -2407,6 +2407,59 @@ paths: "401": *a1 "422": *a2 "429": *a3 + /api/insights/coach/facts: + get: + tags: + - Insights + summary: List the caller's durable Coach facts + description: v1.11.1 — returns the active facts the Coach has extracted about the caller (highest-confidence then newest + first), each decrypted on the fly. The GDPR 'what do you know about me' surface. Coach-gated + (`requireAssistantSurface("coach")`). Auth via cookie or Bearer; the owner is always narrowed from the session, + never the body. Undecryptable rows are omitted rather than failing the read. + responses: + "200": + description: The caller's active facts. + content: + application/json: + schema: + $ref: "#/components/schemas/CoachFactsList" + "401": *a1 + "422": *a2 + "429": *a3 + delete: + tags: + - Insights + summary: Forget all of the caller's Coach facts + description: "v1.11.1 — bulk 'forget what you know about me': soft-deletes every active fact for the caller and returns + the count cleared. Idempotent (a second call clears 0). Coach-gated. Auth via cookie or Bearer." + responses: + "200": + description: All active facts cleared; the count is returned. + content: + application/json: + schema: + $ref: "#/components/schemas/CoachFactsCleared" + "401": *a1 + "422": *a2 + "429": *a3 + /api/insights/coach/facts/{id}: + delete: + tags: + - Insights + summary: Forget one Coach fact + description: "v1.11.1 — soft-deletes a single fact owned by the caller. An unknown / cross-user / already-deleted id is + an idempotent no-op returning `{ deleted: false }`, never revealing whether the id exists under another account. + Coach-gated. Auth via cookie or Bearer." + responses: + "200": + description: "The fact was soft-deleted (`deleted: true`) or the id matched nothing the caller owns (`deleted: false`)." + content: + application/json: + schema: + $ref: "#/components/schemas/CoachFactDeleted" + "401": *a1 + "422": *a2 + "429": *a3 /api/insights/chat/messages/{id}/feedback: post: tags: @@ -7761,6 +7814,114 @@ components: - data - error additionalProperties: false + CoachFactsList: + type: object + properties: + data: + type: object + properties: + facts: + type: array + items: + type: object + properties: + id: + type: string + category: + type: string + enum: + - preference + - condition + - goal + - constraint + - context + description: "App-side closed category: preference | condition | goal | constraint | context." + text: + type: string + description: Decrypted fact text. + confidence: + type: integer + minimum: -9007199254740991 + maximum: 9007199254740991 + description: 0..100 server-assigned extraction confidence. + createdAt: + type: string + format: date-time + pattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d(?::[0-5]\d(?:\.\d+)?)?(?:Z|([+-](?:[01]\d|2[0-3]):[0-5]\d)))$ + required: + - id + - category + - text + - confidence + - createdAt + additionalProperties: false + description: The caller's active facts, highest-confidence then newest first. Undecryptable rows are omitted. + required: + - facts + additionalProperties: false + error: + type: "null" + meta: + type: object + properties: + requestId: + type: string + additionalProperties: false + required: + - data + - error + additionalProperties: false + CoachFactsCleared: + type: object + properties: + data: + type: object + properties: + cleared: + type: integer + minimum: -9007199254740991 + maximum: 9007199254740991 + description: Number of active facts soft-deleted by the bulk clear. + required: + - cleared + additionalProperties: false + error: + type: "null" + meta: + type: object + properties: + requestId: + type: string + additionalProperties: false + required: + - data + - error + additionalProperties: false + CoachFactDeleted: + type: object + properties: + data: + type: object + properties: + deleted: + type: boolean + description: True when a fact owned by the caller was soft-deleted; false for an unknown / cross-user / already-deleted + id (idempotent no-op). + required: + - deleted + additionalProperties: false + error: + type: "null" + meta: + type: object + properties: + requestId: + type: string + additionalProperties: false + required: + - data + - error + additionalProperties: false CoachMessageFeedbackResponse: type: object properties: diff --git a/package.json b/package.json index 525f4bd3e..955eee106 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "healthlog", - "version": "1.11.0", + "version": "1.11.1", "description": "Self-hosted personal-health-tracking PWA with Withings integration, AI insights, and doctor-report PDF export.", "license": "AGPL-3.0-only", "homepage": "https://healthlog.dev", diff --git a/prisma/migrations/0115_v1111_source_aware_rollups/migration.sql b/prisma/migrations/0115_v1111_source_aware_rollups/migration.sql new file mode 100644 index 000000000..68e74729f --- /dev/null +++ b/prisma/migrations/0115_v1111_source_aware_rollups/migration.sql @@ -0,0 +1,32 @@ +-- v1.11.1 — source-aware measurement rollups. +-- +-- `measurement_rollups` is a derived cache; the authoritative readings live in +-- `measurements`. Existing rollup rows are SOURCE-BLIND aggregates: they group +-- by (type, day) only, so two sources reporting the same standard vital for one +-- day (e.g. WHOOP + Apple Watch resting heart rate) were AVG-blended into one +-- row. Re-grain the table to (type, day, source) so the read path can collapse +-- overlapping sources to the ladder-canonical reading while cumulative metrics +-- (steps, energy) still sum per source. +-- +-- The existing rows cannot be re-grained in place (their source is unknown — +-- they already blend N sources), so purge them. This is non-destructive of user +-- data: the boot-time rollup backfill (`rollup-full-backfill`) and the +-- read-time self-heal (`ensureUserRollupsFresh` + the /api/measurements +-- coverage fallback) re-mint per-source rows on next worker boot / first read. +-- Bounded, idempotent, multi-tenant-safe; no operator action required. +DELETE FROM "measurement_rollups"; + +-- Additive column. DEFAULT only to satisfy NOT NULL during the (now empty) +-- table rewrite, then dropped so every writer must supply an explicit source. +ALTER TABLE "measurement_rollups" + ADD COLUMN "source" "measurement_source" NOT NULL DEFAULT 'MANUAL'; +ALTER TABLE "measurement_rollups" ALTER COLUMN "source" DROP DEFAULT; + +-- Re-grain the primary key to include source. The hot descending index +-- (user_id, type, granularity, bucket_start DESC) is unchanged: readers fetch +-- every source for a (user, type) range and collapse in application code, so +-- they never filter by source in SQL. +ALTER TABLE "measurement_rollups" DROP CONSTRAINT "measurement_rollups_pkey"; +ALTER TABLE "measurement_rollups" + ADD CONSTRAINT "measurement_rollups_pkey" + PRIMARY KEY ("user_id", "type", "granularity", "bucket_start", "source"); diff --git a/prisma/migrations/0116_v1111_coach_facts/migration.sql b/prisma/migrations/0116_v1111_coach_facts/migration.sql new file mode 100644 index 000000000..4d56933b1 --- /dev/null +++ b/prisma/migrations/0116_v1111_coach_facts/migration.sql @@ -0,0 +1,27 @@ +-- v1.11.1 — durable Coach personal facts (Epic B, B-W7). +-- +-- Additive, multi-tenant-safe: one new table + index + FK, no column on an +-- existing hot table, no backfill. The fact text is encrypted at rest +-- (AES-256-GCM, same codec as coach_messages.encrypted_content); category / +-- confidence / source stay plain so the injection picker can rank without a +-- per-row decrypt. Soft-delete via deleted_at keeps "forget this" auditable. +CREATE TABLE "coach_facts" ( + "id" TEXT NOT NULL, + "user_id" TEXT NOT NULL, + "fact_encrypted" BYTEA NOT NULL, + "category" TEXT NOT NULL, + "confidence" INTEGER NOT NULL DEFAULT 50, + "source_conversation_id" TEXT, + "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP, + "updated_at" TIMESTAMP(3) NOT NULL, + "deleted_at" TIMESTAMP(3), + + CONSTRAINT "coach_facts_pkey" PRIMARY KEY ("id") +); + +CREATE INDEX "coach_facts_user_id_deleted_at_confidence_idx" + ON "coach_facts" ("user_id", "deleted_at", "confidence"); + +ALTER TABLE "coach_facts" + ADD CONSTRAINT "coach_facts_user_id_fkey" + FOREIGN KEY ("user_id") REFERENCES "users" ("id") ON DELETE CASCADE ON UPDATE CASCADE; diff --git a/prisma/schema.prisma b/prisma/schema.prisma index 232978bfb..358f0c0ad 100644 --- a/prisma/schema.prisma +++ b/prisma/schema.prisma @@ -351,6 +351,7 @@ model User { devices Device[] integrationStatuses IntegrationStatus[] coachConversations CoachConversation[] + coachFacts CoachFact[] coachUsage CoachUsage[] // v1.4.25 W8d — workout passthrough (Apple HKWorkout / Withings // getworkouts / future Strava). Schema lands now so the v1.5 iOS @@ -821,6 +822,13 @@ model MeasurementRollup { type MeasurementType granularity RollupGranularity bucketStart DateTime @map("bucket_start") @db.Timestamptz(3) + /// v1.11.1 — source-aware grain. Rollup rows are minted per source so the + /// read path can collapse overlapping sources (e.g. WHOOP + Apple Watch + /// resting heart rate) to the ladder-canonical reading, while cumulative + /// metrics (steps, energy) sum per source. Always non-null; migration 0115 + /// purged the legacy source-blind aggregates and the boot backfill re-mints + /// per-source rows. + source MeasurementSource count Int mean Float minValue Float @map("min_value") @@ -839,7 +847,7 @@ model MeasurementRollup { user User @relation(fields: [userId], references: [id], onDelete: Cascade) - @@id([userId, type, granularity, bucketStart]) + @@id([userId, type, granularity, bucketStart, source]) @@index([userId, type, granularity, bucketStart(sort: Desc)]) @@map("measurement_rollups") } @@ -2680,6 +2688,37 @@ model CoachMessage { @@map("coach_messages") } +// v1.11.1 (Epic B, B-W7) — durable, user-scoped personal facts the Coach +// extracts from conversations: stable preferences, conditions the user states +// about themselves, goals, constraints, durable life context. The fact TEXT is +// encrypted at rest (AES-256-GCM, the same codec as CoachMessage). `category`, +// `confidence`, and `sourceConversationId` stay plain so the injection picker +// can rank without paying a per-row decrypt. Descriptive, never diagnostic. +// Soft-delete via `deletedAt` so a user "forget this" action is reversible / +// auditable and re-extraction can avoid resurrecting it. +model CoachFact { + id String @id @default(cuid()) + userId String @map("user_id") + factEncrypted Bytes @map("fact_encrypted") + // App-side closed enum (not a DB enum, to keep the migration purely additive + // and avoid an enum-migration per new category): preference | condition | + // goal | constraint | context. + category String + // 0..100 server-assigned extraction confidence. + confidence Int @default(50) + // Provenance — nullable so a future manual-entry surface can leave it null. + sourceConversationId String? @map("source_conversation_id") + createdAt DateTime @default(now()) @map("created_at") + updatedAt DateTime @updatedAt @map("updated_at") + // Soft-delete: hides the fact from injection but keeps the row for audit. + deletedAt DateTime? @map("deleted_at") + + user User @relation(fields: [userId], references: [id], onDelete: Cascade) + + @@index([userId, deletedAt, confidence]) + @@map("coach_facts") +} + // Per-user, per-day token-spend ledger for the AI Coach. The dispatcher // reads `totalTokens` against `MAX_TOKENS_PER_USER_PER_DAY` before // hitting any provider, so a runaway client cannot drain the operator's diff --git a/scripts/rotate-encryption-key.ts b/scripts/rotate-encryption-key.ts index 5ffe0c5de..92c730a50 100644 --- a/scripts/rotate-encryption-key.ts +++ b/scripts/rotate-encryption-key.ts @@ -312,6 +312,82 @@ async function main() { } results.push(coachResult); + // ───── CoachConversation.summaryEncrypted ───── + // v1.11.1 — `Bytes` column carrying the UTF-8 ciphertext of the rolling + // conversation summary. Same Buffer round-trip as CoachMessage; nullable, so + // skip empty/absent rows. + const coachConversations = await prisma.coachConversation.findMany({ + where: { summaryEncrypted: { not: null } }, + select: { id: true, summaryEncrypted: true }, + }); + const summaryResult: RotationResult = { + table: "CoachConversation", + field: "summaryEncrypted", + scanned: coachConversations.length, + rotated: 0, + errors: 0, + }; + for (const row of coachConversations) { + const buf = row.summaryEncrypted; + if (!buf || buf.byteLength === 0) continue; + const asString = Buffer.from(buf).toString("utf8"); + if (!shouldRotate(asString)) continue; + try { + const rotated = encrypt(decrypt(asString)); + const encoded = Buffer.from(rotated, "utf8"); + const next = new Uint8Array(new ArrayBuffer(encoded.byteLength)); + next.set(encoded); + await prisma.coachConversation.update({ + where: { id: row.id }, + data: { summaryEncrypted: next }, + }); + summaryResult.rotated++; + } catch (err) { + summaryResult.errors++; + console.error( + `[CoachConversation.summaryEncrypted] row ${row.id}: ${(err as Error).message}`, + ); + } + } + results.push(summaryResult); + + // ───── CoachFact.factEncrypted ───── + // v1.11.1 — `Bytes` column carrying the UTF-8 ciphertext of each durable + // personal fact. Same Buffer round-trip as CoachMessage. + const coachFacts = await prisma.coachFact.findMany({ + select: { id: true, factEncrypted: true }, + }); + const factResult: RotationResult = { + table: "CoachFact", + field: "factEncrypted", + scanned: coachFacts.length, + rotated: 0, + errors: 0, + }; + for (const row of coachFacts) { + const buf = row.factEncrypted; + if (!buf || buf.byteLength === 0) continue; + const asString = Buffer.from(buf).toString("utf8"); + if (!shouldRotate(asString)) continue; + try { + const rotated = encrypt(decrypt(asString)); + const encoded = Buffer.from(rotated, "utf8"); + const next = new Uint8Array(new ArrayBuffer(encoded.byteLength)); + next.set(encoded); + await prisma.coachFact.update({ + where: { id: row.id }, + data: { factEncrypted: next }, + }); + factResult.rotated++; + } catch (err) { + factResult.errors++; + console.error( + `[CoachFact.factEncrypted] row ${row.id}: ${(err as Error).message}`, + ); + } + } + results.push(factResult); + console.log("\n=== Rotation summary ==="); let totalRotated = 0; let totalErrors = 0; diff --git a/src/app/api/analytics/__tests__/route.test.ts b/src/app/api/analytics/__tests__/route.test.ts index f499922c7..22d4f3a9e 100644 --- a/src/app/api/analytics/__tests__/route.test.ts +++ b/src/app/api/analytics/__tests__/route.test.ts @@ -44,7 +44,12 @@ vi.mock("@/lib/db", () => ({ findFirst: vi.fn().mockResolvedValue(null), deleteMany: vi.fn().mockResolvedValue({ count: 0 }), upsert: vi.fn().mockResolvedValue({}), + // v1.11.1 — the writer now delete-then-inserts via createMany. + createMany: vi.fn().mockResolvedValue({ count: 0 }), }, + // v1.11.1 — the source-aware rollup readers load the user's + // source-priority blob; null → default ladders. + user: { findUnique: vi.fn().mockResolvedValue(null) }, $transaction: vi.fn().mockImplementation(async (queries: unknown[]) => { if (Array.isArray(queries)) return Promise.all(queries); return undefined; @@ -256,8 +261,14 @@ describe("GET /api/analytics", () => { // two parallel ones (all-time + windowed), so the cold fixture // now mocks 4 `$queryRaw` calls: coverage + allTime + windowed + // latest. - vi.mocked(prisma.$queryRaw) - .mockResolvedValueOnce([{ n: BigInt(0) }] as never) + // The coverage probe rides `$queryRaw` (n:0 → cold path). v1.11.1 — the + // cold path's data aggregates (allTime + windowed + latest) now run via + // `$queryRawUnsafe` (the source-aware collapse splices a whitelisted CASE). + vi.mocked(prisma.$queryRaw).mockResolvedValueOnce([ + { n: BigInt(0) }, + ] as never); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + vi.mocked(prisma.$queryRawUnsafe as any) .mockResolvedValueOnce([ { type: "WEIGHT", diff --git a/src/app/api/insights/chat/route.ts b/src/app/api/insights/chat/route.ts index e3e23e92a..bd612c595 100644 --- a/src/app/api/insights/chat/route.ts +++ b/src/app/api/insights/chat/route.ts @@ -55,6 +55,7 @@ import { fetchConversationWithMessages, listConversations, } from "@/lib/ai/coach/persistence"; +import { enqueueCoachMemoryRefresh } from "@/lib/ai/coach/coach-memory-shared"; import { buildDateKey, enforceBudget, @@ -109,23 +110,26 @@ interface CoachTurn { * `{ role, content }` chat shape the provider clients expect. * * Also enforces the 20-turn cap: when the conversation history exceeds - * `TURN_CAP`, the older half is folded into a single synthetic - * "[summary]" user message so the prompt budget stays bounded. This is - * best-effort — we don't pay for a separate provider call to summarise; - * we just keep the last `RECENT_HISTORY` turns verbatim and prepend a - * placeholder line that names the elided count. + * `TURN_CAP`, the older half is folded out of the verbatim window. v1.11.1 — + * if a rolling summary of those elided turns is on file + * (`CoachConversation.summaryEncrypted`, refreshed off-budget by the + * coach-memory-refresh worker) it is prepended so the Coach keeps memory of + * the older conversation; otherwise we fall back to a placeholder that just + * names the elided count (the pre-v1.11.1 behaviour). The summary is read + * stale-while-revalidate — the current turn uses whatever is on disk, the + * enqueued refresh makes the next long turn fresh. */ -function buildHistoryWindow(turns: CoachTurn[]): CoachTurn[] { +function buildHistoryWindow( + turns: CoachTurn[], + summary: string | null, +): CoachTurn[] { if (turns.length <= TURN_CAP) return turns; const elided = turns.length - RECENT_HISTORY; const recent = turns.slice(turns.length - RECENT_HISTORY); - return [ - { - role: "user", - content: `[summary placeholder — ${elided} earlier turns elided to stay within the conversation budget]`, - }, - ...recent, - ]; + const memo = summary + ? `[earlier conversation summary] ${summary}` + : `[summary placeholder — ${elided} earlier turns elided to stay within the conversation budget]`; + return [{ role: "user", content: memo }, ...recent]; } async function handleChatRequest(request: NextRequest): Promise { @@ -197,6 +201,9 @@ async function handleChatRequest(request: NextRequest): Promise { // ── Conversation resolution ────────────────────────────────── let workingConversationId: string; let priorTurns: CoachTurn[] = []; + // v1.11.1 — rolling summary of the elided older turns, read stale-while- + // revalidate; null for a fresh conversation or when none is on file. + let priorSummary: string | null = null; if (conversationId) { const existing = await fetchConversationWithMessages( @@ -208,6 +215,7 @@ async function handleChatRequest(request: NextRequest): Promise { throw new HttpError(404, "coach.conversation.notFound"); } workingConversationId = existing.id; + priorSummary = existing.summary ?? null; priorTurns = existing.messages.map((m) => ({ role: m.role, content: m.content, @@ -286,10 +294,24 @@ async function handleChatRequest(request: NextRequest): Promise { : scope; const snapshot = await buildCoachSnapshot(userId, effectiveScope); const systemPrompt = getCoachSystemPrompt(locale, coachPrefs); - const window = buildHistoryWindow([ + const allTurns: CoachTurn[] = [ ...priorTurns, { role: "user", content: message }, - ]); + ]; + const window = buildHistoryWindow(allTurns, priorSummary); + // v1.11.1 — once a conversation grows past the history cap, refresh the + // rolling summary + extract durable facts off the request path. Fire-and- + // forget: this turn uses whatever summary is already on disk; the refresh + // makes the next long turn fresh. No-ops without an embedded worker. + if (allTurns.length > TURN_CAP) { + void enqueueCoachMemoryRefresh({ + conversationId: workingConversationId, + userId, + // Coach memory prose is composed in de/en only (the snapshot's + // coachLocale); collapse the wider UI locale union here. + locale: locale === "en" ? "en" : "de", + }); + } const transcript = window .map((t) => `${t.role.toUpperCase()}: ${t.content}`) .join("\n\n"); diff --git a/src/app/api/insights/coach/facts/[id]/route.ts b/src/app/api/insights/coach/facts/[id]/route.ts new file mode 100644 index 000000000..e28dd04c7 --- /dev/null +++ b/src/app/api/insights/coach/facts/[id]/route.ts @@ -0,0 +1,57 @@ +/** + * DELETE /api/insights/coach/facts/[id] — soft-delete one Coach fact. + * + * v1.11.1 — the per-fact "forget this one thing" delete for the durable + * Coach facts surface. + * + * Ownership + existence privacy: the soft-delete uses `updateMany` scoped + * `where: { id, userId, deletedAt: null }` rather than `update`. A + * cross-user id, an unknown id, or an already-deleted fact all resolve to + * a `count: 0` no-op — the route returns `200 { deleted: false }` and + * never reveals whether the id exists under another account. The matching + * convention is the idempotent-delete one used elsewhere in the tree + * (a not-found delete is a successful no-op, not a 404). + * + * Coach-gated: same `requireAssistantSurface("coach")` kill-switch as the + * collection route. + */ +import type { NextRequest } from "next/server"; + +import { apiHandler, requireAuth } from "@/lib/api-handler"; +import { apiSuccess } from "@/lib/api-response"; +import { annotate } from "@/lib/logging/context"; +import { prisma } from "@/lib/db"; +import { requireAssistantSurface } from "@/lib/feature-flags"; + +interface RouteCtx { + params: Promise<{ id: string }>; +} + +export const DELETE = apiHandler( + async (_request: NextRequest, ctx: RouteCtx) => { + const { user } = await requireAuth(); + await requireAssistantSurface("coach"); + + const { id } = await ctx.params; + + // `updateMany` (not `update`) so an unknown / cross-user / already-soft- + // deleted id is a 0-count no-op rather than a P2025 throw — the + // existence channel never leaks across accounts. + const { count } = await prisma.coachFact.updateMany({ + where: { id, userId: user.id, deletedAt: null }, + data: { deletedAt: new Date() }, + }); + + const deleted = count > 0; + + annotate({ + action: { name: "coach.facts.deleted" }, + meta: { deleted }, + }); + + return apiSuccess({ deleted }); + }, +); + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; diff --git a/src/app/api/insights/coach/facts/__tests__/route.test.ts b/src/app/api/insights/coach/facts/__tests__/route.test.ts new file mode 100644 index 000000000..6c7cc9f73 --- /dev/null +++ b/src/app/api/insights/coach/facts/__tests__/route.test.ts @@ -0,0 +1,332 @@ +/** + * v1.11.1 — `/api/insights/coach/facts` (+ `/[id]`) routes. + * + * The GDPR / "forget what you know about me" management surface for the + * durable Coach facts. Covers: + * - GET lists only ACTIVE facts (deletedAt: null), decrypted, scoped to + * the caller, highest-confidence-then-newest ordered. + * - GET skips an undecryptable row rather than 500ing the whole list. + * - bulk DELETE soft-deletes all active facts (sets deletedAt) and + * returns the count. + * - [id] DELETE soft-deletes the caller's own fact, and a cross-user / + * unknown id is a 0-count no-op that returns `{ deleted: false }`. + * - the `requireAssistantSurface("coach")` gate is present on every verb. + */ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { NextRequest } from "next/server"; + +vi.mock("@/lib/db", () => ({ + prisma: { + coachFact: { + findMany: vi.fn(), + updateMany: vi.fn(), + }, + }, +})); + +vi.mock("@/lib/auth/session", () => ({ + getSession: vi.fn(), +})); + +// v1.4.31 — `requireAssistantSurface()` gates near the top of each +// handler. Mock the module boundary so flag reads are deterministic; the +// gate-presence assertions below verify the call is actually made. +vi.mock("@/lib/feature-flags", () => ({ + requireAssistantSurface: vi.fn(async () => undefined), + AssistantDisabledError: class extends Error {}, +})); + +// Mock the codec so the test never needs an encryption key. The "row" +// carries a tagged Uint8Array and the mock maps it back to a string; +// a sentinel buffer triggers a throw to exercise the fail-closed skip. +vi.mock("@/lib/ai/coach/bytes-codec", () => ({ + decryptFromBytes: vi.fn((buf: Uint8Array) => { + const tag = Buffer.from(buf).toString("utf8"); + if (tag === "__undecryptable__") { + throw new Error("unknown key id"); + } + return `decrypted:${tag}`; + }), +})); + +vi.mock("@/lib/logging/transports", () => ({ + emitIfSampled: vi.fn(), +})); + +vi.mock("@/lib/db-compat", () => ({ + ensureDbCompatibility: vi.fn().mockResolvedValue(undefined), +})); + +vi.mock("@/lib/logging/context", async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + annotate: vi.fn(), + }; +}); + +vi.mock("next/headers", () => ({ + headers: vi.fn(async () => ({ get: () => null })), + cookies: vi.fn(async () => ({ + get: () => undefined, + set: () => {}, + delete: () => {}, + })), +})); + +import { GET, DELETE } from "../route"; +import { DELETE as DELETE_ONE } from "../[id]/route"; +import { prisma } from "@/lib/db"; +import { getSession } from "@/lib/auth/session"; +import { requireAssistantSurface } from "@/lib/feature-flags"; +import { annotate } from "@/lib/logging/context"; + +const SESSION_OK = { + session: { id: "sess-1", expiresAt: new Date(Date.now() + 3_600_000) }, + user: { + id: "user-1", + username: "tester", + role: "USER" as const, + displayName: null, + }, +}; + +const callGet = GET as unknown as () => Promise; +const callDeleteAll = DELETE as unknown as () => Promise; +const callDeleteOne = DELETE_ONE as unknown as ( + req: NextRequest, + ctx: { params: Promise<{ id: string }> }, +) => Promise; + +function bytes(tag: string): Uint8Array { + return new Uint8Array(Buffer.from(tag, "utf8")); +} + +function deleteReq(): NextRequest { + return new NextRequest("http://localhost/api/insights/coach/facts/x", { + method: "DELETE", + }); +} + +beforeEach(() => { + vi.resetAllMocks(); + vi.mocked(getSession).mockResolvedValue(SESSION_OK as never); + vi.mocked(requireAssistantSurface).mockResolvedValue(undefined as never); +}); + +describe("GET /api/insights/coach/facts", () => { + it("lists the caller's active facts, decrypted, scoped to the user", async () => { + vi.mocked(prisma.coachFact.findMany).mockResolvedValue([ + { + id: "f1", + category: "goal", + factEncrypted: bytes("lose 5kg"), + confidence: 90, + createdAt: new Date("2026-06-01T00:00:00Z"), + }, + { + id: "f2", + category: "preference", + factEncrypted: bytes("morning workouts"), + confidence: 70, + createdAt: new Date("2026-06-02T00:00:00Z"), + }, + ] as never); + + const res = await callGet(); + expect(res.status).toBe(200); + const body = (await res.json()) as { + data: { + facts: Array<{ + id: string; + category: string; + text: string; + confidence: number; + }>; + }; + }; + expect(body.data.facts).toHaveLength(2); + expect(body.data.facts[0]).toMatchObject({ + id: "f1", + category: "goal", + text: "decrypted:lose 5kg", + confidence: 90, + }); + expect(body.data.facts[1]?.text).toBe("decrypted:morning workouts"); + + // Ownership-scoped + active-only query. + const where = vi.mocked(prisma.coachFact.findMany).mock.calls[0]?.[0] + ?.where as { userId: string; deletedAt: null }; + expect(where.userId).toBe("user-1"); + expect(where.deletedAt).toBeNull(); + }); + + it("orders highest-confidence then newest", async () => { + vi.mocked(prisma.coachFact.findMany).mockResolvedValue([] as never); + await callGet(); + const orderBy = vi.mocked(prisma.coachFact.findMany).mock.calls[0]?.[0] + ?.orderBy; + expect(orderBy).toEqual([ + { confidence: "desc" }, + { createdAt: "desc" }, + ]); + }); + + it("skips an undecryptable row rather than 500ing the whole list", async () => { + vi.mocked(prisma.coachFact.findMany).mockResolvedValue([ + { + id: "f1", + category: "goal", + factEncrypted: bytes("__undecryptable__"), + confidence: 90, + createdAt: new Date("2026-06-01T00:00:00Z"), + }, + { + id: "f2", + category: "context", + factEncrypted: bytes("works night shifts"), + confidence: 60, + createdAt: new Date("2026-06-02T00:00:00Z"), + }, + ] as never); + + const res = await callGet(); + expect(res.status).toBe(200); + const body = (await res.json()) as { + data: { facts: Array<{ id: string; text: string }> }; + }; + // The undecryptable row is dropped; the good one survives. + expect(body.data.facts).toHaveLength(1); + expect(body.data.facts[0]?.id).toBe("f2"); + expect(body.data.facts[0]?.text).toBe("decrypted:works night shifts"); + }); + + it("annotates coach.facts.listed with the count only", async () => { + vi.mocked(prisma.coachFact.findMany).mockResolvedValue([ + { + id: "f1", + category: "goal", + factEncrypted: bytes("a"), + confidence: 50, + createdAt: new Date(), + }, + ] as never); + + await callGet(); + const call = vi + .mocked(annotate) + .mock.calls.find( + (c) => + (c[0] as { action?: { name?: string } })?.action?.name === + "coach.facts.listed", + ); + expect(call).toBeTruthy(); + expect((call![0] as { meta?: { count?: number } }).meta).toEqual({ + count: 1, + }); + }); + + it("invokes the coach assistant-surface gate", async () => { + vi.mocked(prisma.coachFact.findMany).mockResolvedValue([] as never); + await callGet(); + expect(requireAssistantSurface).toHaveBeenCalledWith("coach"); + }); +}); + +describe("DELETE /api/insights/coach/facts — forget all", () => { + it("soft-deletes all active facts and returns the count", async () => { + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 3, + } as never); + + const res = await callDeleteAll(); + expect(res.status).toBe(200); + const body = (await res.json()) as { data: { cleared: number } }; + expect(body.data.cleared).toBe(3); + + const arg = vi.mocked(prisma.coachFact.updateMany).mock.calls[0]?.[0] as { + where: { userId: string; deletedAt: null }; + data: { deletedAt: Date }; + }; + expect(arg.where.userId).toBe("user-1"); + expect(arg.where.deletedAt).toBeNull(); + // Soft-delete sets a deletedAt timestamp; the row is not removed. + expect(arg.data.deletedAt).toBeInstanceOf(Date); + }); + + it("is idempotent — a second clear returns 0", async () => { + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 0, + } as never); + const res = await callDeleteAll(); + expect(res.status).toBe(200); + const body = (await res.json()) as { data: { cleared: number } }; + expect(body.data.cleared).toBe(0); + }); + + it("invokes the coach assistant-surface gate", async () => { + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 0, + } as never); + await callDeleteAll(); + expect(requireAssistantSurface).toHaveBeenCalledWith("coach"); + }); +}); + +describe("DELETE /api/insights/coach/facts/[id] — forget one", () => { + it("soft-deletes the caller's own fact and returns deleted:true", async () => { + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 1, + } as never); + + const res = await callDeleteOne(deleteReq(), { + params: Promise.resolve({ id: "f1" }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { data: { deleted: boolean } }; + expect(body.data.deleted).toBe(true); + + const arg = vi.mocked(prisma.coachFact.updateMany).mock.calls[0]?.[0] as { + where: { id: string; userId: string; deletedAt: null }; + data: { deletedAt: Date }; + }; + // Scoped by BOTH id and userId, active-only. + expect(arg.where.id).toBe("f1"); + expect(arg.where.userId).toBe("user-1"); + expect(arg.where.deletedAt).toBeNull(); + expect(arg.data.deletedAt).toBeInstanceOf(Date); + }); + + it("returns deleted:false for an unknown / cross-user id (0-count no-op)", async () => { + // updateMany scoped to { id, userId } matches no row owned by the + // caller → count 0 → no other user's row is ever touched. + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 0, + } as never); + + const res = await callDeleteOne(deleteReq(), { + params: Promise.resolve({ id: "someone-elses-fact" }), + }); + expect(res.status).toBe(200); + const body = (await res.json()) as { data: { deleted: boolean } }; + expect(body.data.deleted).toBe(false); + + // The where clause pins userId so the query can never reach a + // foreign row regardless of the id supplied. + const arg = vi.mocked(prisma.coachFact.updateMany).mock.calls[0]?.[0] as { + where: { userId: string }; + }; + expect(arg.where.userId).toBe("user-1"); + }); + + it("invokes the coach assistant-surface gate", async () => { + vi.mocked(prisma.coachFact.updateMany).mockResolvedValue({ + count: 0, + } as never); + await callDeleteOne(deleteReq(), { + params: Promise.resolve({ id: "f1" }), + }); + expect(requireAssistantSurface).toHaveBeenCalledWith("coach"); + }); +}); diff --git a/src/app/api/insights/coach/facts/route.ts b/src/app/api/insights/coach/facts/route.ts new file mode 100644 index 000000000..4a37ce2ef --- /dev/null +++ b/src/app/api/insights/coach/facts/route.ts @@ -0,0 +1,102 @@ +/** + * GET /api/insights/coach/facts — list the caller's ACTIVE Coach facts. + * DELETE /api/insights/coach/facts — "forget all": soft-delete every active + * fact for the caller. + * + * v1.11.1 — the GDPR / "forget what you know about me" surface for the + * durable Coach facts the assistant extracts during conversations. Facts + * are server-extracted, not user-authored, so this surface is read + + * delete only (no POST / PATCH). + * + * Ownership: every query is scoped `where: { userId, ... }`, so a caller + * can only ever see or clear their own facts. The fact text is decrypted + * on the fly; an undecryptable row (e.g. a key rotated out of the map) is + * skipped rather than 500ing the whole list — the surface stays available + * for the rows that DO decrypt. + * + * Coach-gated: a fact only exists because the Coach extracted it, so the + * management surface sits behind the same `requireAssistantSurface("coach")` + * kill-switch as the rest of the Coach stack. + */ +import { apiHandler, requireAuth } from "@/lib/api-handler"; +import { apiSuccess } from "@/lib/api-response"; +import { annotate } from "@/lib/logging/context"; +import { prisma } from "@/lib/db"; +import { requireAssistantSurface } from "@/lib/feature-flags"; +import { decryptFromBytes } from "@/lib/ai/coach/bytes-codec"; + +export const GET = apiHandler(async () => { + const { user } = await requireAuth(); + await requireAssistantSurface("coach"); + + const rows = await prisma.coachFact.findMany({ + where: { userId: user.id, deletedAt: null }, + // Highest-confidence first, then newest — mirrors the injection + // ordering so the management list reads in the same priority the + // assistant actually weights the facts. + orderBy: [{ confidence: "desc" }, { createdAt: "desc" }], + select: { + id: true, + category: true, + factEncrypted: true, + confidence: true, + createdAt: true, + }, + }); + + const facts: Array<{ + id: string; + category: string; + text: string; + confidence: number; + createdAt: Date; + }> = []; + + for (const row of rows) { + let text: string; + try { + text = decryptFromBytes(row.factEncrypted); + } catch { + // Fail closed per row — never surface ciphertext, never 500 the + // whole list because one row's key id is no longer in the map. + continue; + } + facts.push({ + id: row.id, + category: row.category, + text, + confidence: row.confidence, + createdAt: row.createdAt, + }); + } + + annotate({ + action: { name: "coach.facts.listed" }, + meta: { count: facts.length }, + }); + + return apiSuccess({ facts }); +}); + +export const DELETE = apiHandler(async () => { + const { user } = await requireAuth(); + await requireAssistantSurface("coach"); + + // Soft-delete every active fact — keeps the rows for audit while + // hiding them from injection. `updateMany` scoped to the caller can + // never touch another user's rows. + const { count } = await prisma.coachFact.updateMany({ + where: { userId: user.id, deletedAt: null }, + data: { deletedAt: new Date() }, + }); + + annotate({ + action: { name: "coach.facts.cleared" }, + meta: { cleared: count }, + }); + + return apiSuccess({ cleared: count }); +}); + +export const dynamic = "force-dynamic"; +export const runtime = "nodejs"; diff --git a/src/app/api/measurements/__tests__/range-aggregation-route.test.ts b/src/app/api/measurements/__tests__/range-aggregation-route.test.ts index fddbbb06e..6f92b73eb 100644 --- a/src/app/api/measurements/__tests__/range-aggregation-route.test.ts +++ b/src/app/api/measurements/__tests__/range-aggregation-route.test.ts @@ -23,6 +23,11 @@ vi.mock("@/lib/db", () => ({ measurementRollup: { findMany: vi.fn(), }, + // v1.11.1 — the rollup path calls `loadUserSourcePriority` + // (prisma.user.findUnique) to build the source-rank ladders before + // `collapseRollupRowsBySource` runs. This is a separate read from the + // date_trunc `$queryRaw`; `null` → default ladders. + user: { findUnique: vi.fn() }, $queryRaw: vi.fn(), }, })); @@ -77,6 +82,9 @@ function getRequest(query: string): NextRequest { beforeEach(() => { vi.resetAllMocks(); vi.mocked(getSession).mockResolvedValue(SESSION_OK as never); + // v1.11.1 — null source-priority blob → default rank ladders for the + // rollup-path source collapse. + vi.mocked(prisma.user.findUnique).mockResolvedValue(null as never); }); describe("GET /api/measurements — aggregation gate (C2)", () => { @@ -223,6 +231,8 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { bucketStart: new Date(Date.UTC(2026, 3, 16 + i, 0, 0, 0)), mean: 81 + i * 0.05, count: 5, + // v1.11.1 — single source per day → collapse is identity. + source: "APPLE_HEALTH", })); vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( buckets as never, @@ -261,6 +271,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 2000, // would yield 10000 if multiplied by count=5 count: 5, sumValue: 11000 + i * 250, // distinct from mean × count + source: "APPLE_HEALTH", })); vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( buckets as never, @@ -292,6 +303,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 2000, count: 5, sumValue: null, + source: "APPLE_HEALTH", }, ]; vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( @@ -354,6 +366,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 128, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, { type: "BLOOD_PRESSURE_SYS", @@ -361,6 +374,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 132, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, ]; const convergedRollup = Array.from({ length: 28 }, (_, i) => ({ @@ -369,6 +383,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 128 + (i % 5), count: 2, sumValue: null, + source: "APPLE_HEALTH", })); // First read returns the sparse rows; second (post-fold) read // returns the converged rows. @@ -416,6 +431,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 81 + i * 0.05, count: 2, sumValue: null, + source: "APPLE_HEALTH", })); vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( buckets as never, @@ -444,6 +460,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 128, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, { type: "BLOOD_PRESSURE_SYS", @@ -451,6 +468,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 132, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, ]; vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( @@ -484,6 +502,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 128, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, { type: "BLOOD_PRESSURE_SYS", @@ -491,6 +510,7 @@ describe("GET /api/measurements — all-time semantics (SD-H1)", () => { mean: 132, count: 1, sumValue: null, + source: "APPLE_HEALTH", }, ]; vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue( diff --git a/src/app/api/measurements/route.ts b/src/app/api/measurements/route.ts index ce3a032fb..11c5939b3 100644 --- a/src/app/api/measurements/route.ts +++ b/src/app/api/measurements/route.ts @@ -34,6 +34,11 @@ import { collapseToTypeDayKeys, recomputeUserRollups, } from "@/lib/rollups/measurement-rollups"; +import { + collapseRollupRowsBySource, + loadUserSourcePriority, +} from "@/lib/rollups/measurement-read"; +import { buildSourceRankCase } from "@/lib/analytics/source-rank-sql"; import { NextRequest } from "next/server"; import type { MeasurementType, @@ -308,30 +313,40 @@ export const GET = apiHandler(async (request: NextRequest) => { to ) { const cap = Math.min(limit, BUCKET_CAP.daily); - const rollupRows = await prisma.measurementRollup.findMany({ - where: { - userId: user.id, - type: type as MeasurementType, - granularity: "DAY", - bucketStart: { gte: from, lte: to }, - }, - orderBy: { bucketStart: "asc" }, - take: cap, - select: { - type: true, - bucketStart: true, - mean: true, - count: true, - // v1.4.39 W-SUM — the writer now populates `sum_value` on - // every fold. The cumulative metric path consumes the column - // directly instead of reconstructing `mean * count`; the - // legacy fallback below covers the boot-backfill convergence - // window when an existing bucket pre-dates v1.4.39. - sumValue: true, - minValue: true, - maxValue: true, - }, - }); + // v1.11.1 — rollup rows are per source; collapse overlapping sources to the + // ladder-canonical reading per day before the chart consumes them, so a + // dual-source vital paints one line, not two stacked points per day. + const priorityJson = await loadUserSourcePriority(user.id); + const rollupRows = collapseRollupRowsBySource( + await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: type as MeasurementType, + granularity: "DAY", + bucketStart: { gte: from, lte: to }, + }, + orderBy: { bucketStart: "asc" }, + // Fetch beyond the bucket cap pre-collapse — N sources per day can + // exceed `cap` rows before the per-day collapse reduces them. + select: { + type: true, + source: true, + bucketStart: true, + mean: true, + count: true, + // v1.4.39 W-SUM — the writer now populates `sum_value` on + // every fold. The cumulative metric path consumes the column + // directly instead of reconstructing `mean * count`; the + // legacy fallback below covers the boot-backfill convergence + // window when an existing bucket pre-dates v1.4.39. + sumValue: true, + minValue: true, + maxValue: true, + }, + }), + type as MeasurementType, + priorityJson, + ).slice(0, cap); // v1.4.39.2 — coverage-mismatch fallback. v1.4.39.1 wired the // rollup write hook into the previously-bypassed Withings sync, // /api/import, and admin-restore paths, and widened the boot-time @@ -403,26 +418,31 @@ export const GET = apiHandler(async (request: NextRequest) => { }); // Re-read so this request returns the converged rows. The // next chart paint hits the same warm rollup without paying - // the fallback cost. - effectiveRows = await prisma.measurementRollup.findMany({ - where: { - userId: user.id, - type: type as MeasurementType, - granularity: "DAY", - bucketStart: { gte: from, lte: to }, - }, - orderBy: { bucketStart: "asc" }, - take: cap, - select: { - type: true, - bucketStart: true, - mean: true, - count: true, - sumValue: true, - minValue: true, - maxValue: true, - }, - }); + // the fallback cost. v1.11.1 — collapse per-source rows to the + // canonical reading per day, same as the initial read. + effectiveRows = collapseRollupRowsBySource( + await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: type as MeasurementType, + granularity: "DAY", + bucketStart: { gte: from, lte: to }, + }, + orderBy: { bucketStart: "asc" }, + select: { + type: true, + source: true, + bucketStart: true, + mean: true, + count: true, + sumValue: true, + minValue: true, + maxValue: true, + }, + }), + type as MeasurementType, + priorityJson, + ).slice(0, cap); } catch (err) { // Best-effort — fall back to the legacy live aggregate path // below if the fold itself failed so the chart paints @@ -520,20 +540,48 @@ export const GET = apiHandler(async (request: NextRequest) => { const aggregator = useSum ? Prisma.raw(`SUM(m."value")::double precision`) : Prisma.raw(`AVG(m."value")::double precision`); + // v1.11.1 — collapse overlapping sources to the ladder-canonical reading + // per (type, bucket) before aggregating, so this cold/uncovered fallback + // agrees with the warm rollup path. `canon` picks the winning source for + // each (type, bucket) via the per-user rank; the join keeps only its rows. + const priorityJson = await loadUserSourcePriority(user.id); + const rankRaw = Prisma.raw( + buildSourceRankCase(priorityJson, 'm."type"', 'm."source"'), + ); + const typeFilter = type + ? Prisma.sql`AND m."type" = ${type}::measurement_type` + : Prisma.empty; const buckets = await prisma.$queryRaw< Array<{ type: string; bucket_start: Date; avg: number; cnt: number }> >` + WITH canon AS ( + SELECT DISTINCT ON (m."type", date_trunc(${truncUnitLiteral}, m."measured_at")) + m."type" AS t, + date_trunc(${truncUnitLiteral}, m."measured_at") AS d, + m."source" AS canon + FROM measurements m + WHERE m."user_id" = ${user.id} + AND m."measured_at" >= ${from} + AND m."measured_at" <= ${to} + AND m."deleted_at" IS NULL + ${typeFilter} + ORDER BY m."type", date_trunc(${truncUnitLiteral}, m."measured_at"), (${rankRaw}), m."source" + ) SELECT m."type"::text AS type, date_trunc(${truncUnitLiteral}, m."measured_at") AS bucket_start, ${aggregator} AS avg, COUNT(*)::int AS cnt FROM measurements m + JOIN canon c + ON c.t = m."type" + AND c.d = date_trunc(${truncUnitLiteral}, m."measured_at") + AND c.canon = m."source" WHERE m."user_id" = ${user.id} AND m."measured_at" >= ${from} AND m."measured_at" <= ${to} AND m."deleted_at" IS NULL - ${type ? Prisma.sql`AND m."type" = ${type}::measurement_type` : Prisma.empty} + ${typeFilter} GROUP BY m."type", bucket_start ORDER BY bucket_start ASC LIMIT ${cap} diff --git a/src/lib/ai/coach/__tests__/conversation-summary.test.ts b/src/lib/ai/coach/__tests__/conversation-summary.test.ts new file mode 100644 index 000000000..7791a7a1e --- /dev/null +++ b/src/lib/ai/coach/__tests__/conversation-summary.test.ts @@ -0,0 +1,210 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import type { StatusProviderResult } from "@/lib/insights/status-provider"; + +// Mock the crypto codec so the unit test needs no encryption key in env. +// `decryptFromBytes` echoes the bytes back as a marker string so the merge +// path can be asserted; `encryptToBytes` is a spy that returns a sentinel. +vi.mock("../bytes-codec", () => ({ + encryptToBytes: vi.fn((plaintext: string) => ({ + __enc: plaintext, + })), + decryptFromBytes: vi.fn((buf: { __plain?: string }) => buf.__plain ?? ""), +})); + +import { encryptToBytes } from "../bytes-codec"; +import { + buildSummaryUserPrompt, + refreshConversationSummary, + SUMMARY_REFRESH_TURN_DELTA, +} from "../conversation-summary"; + +const TURN_CAP = 20; +const RECENT_HISTORY = 18; + +/** A persisted message row as the module's select shape returns it. */ +function msg(role: "user" | "assistant", content: string) { + return { role, encryptedContent: { __plain: content } }; +} + +/** Build a conversation row of `n` turns alternating user/assistant. */ +function makeConversation( + n: number, + overrides: { + summaryEncrypted?: unknown; + summaryTurnCount?: number; + } = {}, +) { + const messages = Array.from({ length: n }, (_, i) => + msg(i % 2 === 0 ? "user" : "assistant", `turn-${i}`), + ); + return { + id: "conv-1", + summaryEncrypted: overrides.summaryEncrypted ?? null, + summaryTurnCount: overrides.summaryTurnCount ?? 0, + messages, + }; +} + +function makePrisma(conversation: unknown) { + const update = vi.fn().mockResolvedValue({}); + const findFirst = vi.fn().mockResolvedValue(conversation); + return { + update, + findFirst, + client: { + coachConversation: { findFirst, update }, + } as never, + }; +} + +const okCompletion: StatusProviderResult = { + kind: "ok", + content: "The user is training for a 10k and prefers morning runs.", + providerType: "admin-openai", + model: "gpt-test", + tokensUsed: 42, +}; + +beforeEach(() => { + vi.clearAllMocks(); +}); + +describe("refreshConversationSummary", () => { + it("(a) returns insufficient with no provider call at or below TURN_CAP", async () => { + const { client, update } = makePrisma(makeConversation(TURN_CAP)); + const runCompletion = vi.fn(); + + const result = await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion, + }); + + expect(result.status).toBe("insufficient"); + expect(runCompletion).not.toHaveBeenCalled(); + expect(update).not.toHaveBeenCalled(); + }); + + it("returns insufficient when the conversation does not exist / is not owned", async () => { + const { client, update } = makePrisma(null); + const runCompletion = vi.fn(); + + const result = await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion, + }); + + expect(result.status).toBe("insufficient"); + expect(runCompletion).not.toHaveBeenCalled(); + expect(update).not.toHaveBeenCalled(); + }); + + it("(b) returns fresh with no provider call when fewer than DELTA new turns accumulated", async () => { + // 28 turns → foldHighWater = 28 - 18 = 10. summaryTurnCount = 10 - (DELTA-1) + // is > 0, and delta = DELTA-1 < DELTA ⇒ fresh. + const foldHighWater = 28 - RECENT_HISTORY; + const { client, update } = makePrisma( + makeConversation(28, { + summaryTurnCount: foldHighWater - (SUMMARY_REFRESH_TURN_DELTA - 1), + }), + ); + const runCompletion = vi.fn(); + + const result = await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion, + }); + + expect(result.status).toBe("fresh"); + expect(runCompletion).not.toHaveBeenCalled(); + expect(update).not.toHaveBeenCalled(); + }); + + it("(c) merge path passes the decrypted prior summary into the user prompt", async () => { + const prior = "Earlier: user wants better sleep and dislikes evening caffeine."; + // 30 turns, no prior summaryTurnCount → forces a generation. + const { client } = makePrisma( + makeConversation(30, { + summaryEncrypted: { __plain: prior }, + summaryTurnCount: 0, + }), + ); + const runCompletion = vi + .fn<(args: { userPrompt: string }) => Promise>() + .mockResolvedValue(okCompletion); + + await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion: runCompletion as never, + }); + + expect(runCompletion).toHaveBeenCalledTimes(1); + const arg = runCompletion.mock.calls[0]![0]; + expect(arg.userPrompt).toContain(prior); + expect(arg.userPrompt).toContain("PRIOR SUMMARY"); + }); + + it("(d) timeout / none leaves the existing summary untouched (skipped, no update)", async () => { + for (const kind of ["timeout", "none", "error"] as const) { + vi.clearAllMocks(); + const { client, update } = makePrisma(makeConversation(30)); + const runCompletion = vi.fn().mockResolvedValue({ kind }); + + const result = await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion, + }); + + expect(result.status).toBe("skipped"); + expect(update).not.toHaveBeenCalled(); + } + }); + + it("(e) success path encrypts and persists summaryEncrypted + turn count + updatedAt", async () => { + const now = new Date("2026-06-04T08:00:00.000Z"); + const { client, update } = makePrisma(makeConversation(30, { summaryTurnCount: 0 })); + const runCompletion = vi.fn().mockResolvedValue(okCompletion); + + const result = await refreshConversationSummary("conv-1", "user-1", { + prisma: client, + runCompletion, + now, + }); + + expect(result.status).toBe("generated"); + expect(encryptToBytes).toHaveBeenCalledWith(okCompletion.content); + expect(update).toHaveBeenCalledTimes(1); + const updateArg = update.mock.calls[0]![0]; + expect(updateArg.where).toEqual({ id: "conv-1" }); + expect(updateArg.data.summaryEncrypted).toBeDefined(); + expect(updateArg.data.summaryUpdatedAt).toBe(now); + // foldHighWater = 30 - 18 = 12. + expect(updateArg.data.summaryTurnCount).toBe(12); + }); +}); + +describe("buildSummaryUserPrompt", () => { + it("uses (none) when no prior summary is supplied", () => { + const out = buildSummaryUserPrompt(null, [ + { role: "user", content: "hi" }, + ]); + expect(out).toContain("PRIOR SUMMARY\n(none)"); + expect(out).toContain("user: hi"); + }); + + it("includes the prior summary and role-prefixes each folded turn", () => { + const out = buildSummaryUserPrompt("prior text", [ + { role: "user", content: "a" }, + { role: "assistant", content: "b" }, + ]); + expect(out).toContain("PRIOR SUMMARY\nprior text"); + expect(out).toContain("user: a"); + expect(out).toContain("assistant: b"); + }); + + it("renders German labels when locale is de", () => { + const out = buildSummaryUserPrompt(null, [], "de"); + expect(out).toContain("FRÜHERE ZUSAMMENFASSUNG"); + expect(out).toContain("(keine)"); + }); +}); diff --git a/src/lib/ai/coach/__tests__/facts-block.test.ts b/src/lib/ai/coach/__tests__/facts-block.test.ts new file mode 100644 index 000000000..dd53eeb20 --- /dev/null +++ b/src/lib/ai/coach/__tests__/facts-block.test.ts @@ -0,0 +1,114 @@ +import { describe, expect, it, vi } from "vitest"; + +// Hermetic codec — same as facts.test.ts. +vi.mock("../bytes-codec", () => ({ + encryptToBytes: (s: string) => new TextEncoder().encode(s), + decryptFromBytes: (b: Uint8Array) => new TextDecoder().decode(b), +})); + +vi.mock("@/lib/db", () => ({ prisma: {} })); + +vi.mock("@/lib/insights/status-provider", () => ({ + runStatusCompletion: vi.fn(), +})); + +import { buildCoachFactsBlock, FACTS_INJECT_TOP_N } from "../facts"; + +function bytes(s: string): Uint8Array { + return new TextEncoder().encode(s); +} + +interface SeedRow { + text: string; + category?: string; + confidence: number; + updatedAt: Date; +} + +/** + * Fake prisma that honours the orderBy the helper passes (confidence DESC, + * updatedAt DESC) so the test exercises the real ranking contract. + */ +function makeFakePrisma(seed: SeedRow[]) { + const findMany = vi.fn(async () => { + const sorted = [...seed].sort((a, b) => { + if (b.confidence !== a.confidence) return b.confidence - a.confidence; + return b.updatedAt.getTime() - a.updatedAt.getTime(); + }); + return sorted.map((r) => ({ + factEncrypted: bytes(r.text), + category: r.category ?? "preference", + confidence: r.confidence, + updatedAt: r.updatedAt, + })); + }); + return { prisma: { coachFact: { findMany } }, findMany }; +} + +describe("buildCoachFactsBlock", () => { + it("ranks by confidence then recency", async () => { + const { prisma } = makeFakePrisma([ + { text: "older high", confidence: 90, updatedAt: new Date(2026, 0, 1) }, + { text: "newer high", confidence: 90, updatedAt: new Date(2026, 0, 5) }, + { text: "low", confidence: 40, updatedAt: new Date(2026, 0, 10) }, + ]); + + const block = await buildCoachFactsBlock("user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + }); + + expect(block).not.toBeNull(); + expect(block!.facts.map((f) => f.text)).toEqual(["newer high", "older high", "low"]); + }); + + it("takes only the top N", async () => { + const seed: SeedRow[] = Array.from({ length: FACTS_INJECT_TOP_N + 4 }, (_, i) => ({ + text: `fact ${i}`, + confidence: 100 - i, + updatedAt: new Date(2026, 0, 1 + i), + })); + const { prisma } = makeFakePrisma(seed); + + const block = await buildCoachFactsBlock("user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + }); + + expect(block).not.toBeNull(); + expect(block!.facts).toHaveLength(FACTS_INJECT_TOP_N); + expect(block!.facts[0].text).toBe("fact 0"); + }); + + it("returns null when there are no active facts", async () => { + const { prisma } = makeFakePrisma([]); + const block = await buildCoachFactsBlock("user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + }); + expect(block).toBeNull(); + }); + + it("skips undecryptable rows fault-isolated (never throws)", async () => { + // One row whose codec throws on decrypt; the helper must skip it. + const findMany = vi.fn(async () => [ + { factEncrypted: bytes("good fact"), category: "goal", confidence: 80, updatedAt: new Date() }, + { factEncrypted: bytes("__throw__"), category: "goal", confidence: 70, updatedAt: new Date() }, + ]); + // Re-point the codec mock to throw for the sentinel payload. + const codec = await import("../bytes-codec"); + vi.spyOn(codec, "decryptFromBytes").mockImplementation((b: Uint8Array) => { + const s = new TextDecoder().decode(b); + if (s === "__throw__") throw new Error("bad key id"); + return s; + }); + + const block = await buildCoachFactsBlock("user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: { coachFact: { findMany } } as any, + }); + + expect(block).not.toBeNull(); + expect(block!.facts.map((f) => f.text)).toEqual(["good fact"]); + }); +}); diff --git a/src/lib/ai/coach/__tests__/facts.test.ts b/src/lib/ai/coach/__tests__/facts.test.ts new file mode 100644 index 000000000..0436dc38e --- /dev/null +++ b/src/lib/ai/coach/__tests__/facts.test.ts @@ -0,0 +1,294 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import type { StatusProviderResult } from "@/lib/insights/status-provider"; + +// Hermetic codec: round-trip through a UTF-8 buffer, no real crypto keys +// needed. Mirrors the bytes-codec contract (string ↔ Uint8Array). +vi.mock("../bytes-codec", () => ({ + encryptToBytes: (s: string) => new TextEncoder().encode(s), + decryptFromBytes: (b: Uint8Array) => new TextDecoder().decode(b), +})); + +// Avoid importing the real Prisma client; opts injection supplies the fake. +vi.mock("@/lib/db", () => ({ prisma: {} })); + +const annotateMock = vi.fn(); +vi.mock("@/lib/logging/context", () => ({ + annotate: (...args: unknown[]) => annotateMock(...args), +})); + +// status-provider is replaced by the injected runCompletion in every test, +// but the module imports it at load time. +vi.mock("@/lib/insights/status-provider", () => ({ + runStatusCompletion: vi.fn(), +})); + +import { + extractAndStoreFacts, + FACT_MAX_CHARS, + MAX_FACTS_PER_USER, +} from "../facts"; + +// --------------------------------------------------------------------------- +// Fake prisma builder +// --------------------------------------------------------------------------- + +interface FakeFactRow { + id: string; + factEncrypted: Uint8Array; + category: string; + confidence: number; + updatedAt: Date; + deletedAt: Date | null; +} + +function bytes(s: string): Uint8Array { + return new TextEncoder().encode(s); +} + +function makeFakePrisma(opts: { + activeFacts?: Array<{ id?: string; text: string; category?: string; confidence?: number }>; + turns?: Array<{ role: string; content: string }>; + conversationExists?: boolean; +}) { + const rows: FakeFactRow[] = (opts.activeFacts ?? []).map((f, i) => ({ + id: f.id ?? `fact-${i}`, + factEncrypted: bytes(f.text), + category: f.category ?? "preference", + confidence: f.confidence ?? 50, + updatedAt: new Date(2026, 0, 1 + i), + deletedAt: null, + })); + + const createCalls: Array> = []; + + const messages = (opts.turns ?? []).map((t) => ({ + role: t.role, + encryptedContent: bytes(t.content), + createdAt: new Date(), + })); + + const prisma = { + coachFact: { + findMany: vi.fn(async () => rows), + create: vi.fn(async (arg: { data: Record }) => { + createCalls.push(arg.data); + return { id: `new-${createCalls.length}`, ...arg.data }; + }), + }, + coachConversation: { + findFirst: vi.fn(async () => + opts.conversationExists === false ? null : { messages }, + ), + }, + }; + + return { prisma, createCalls }; +} + +function ok(content: string): StatusProviderResult { + return { kind: "ok", content, providerType: "mock", model: "m", tokensUsed: 1 }; +} + +beforeEach(() => { + annotateMock.mockClear(); +}); + +describe("extractAndStoreFacts", () => { + it("(a) parses durable facts and persists them field-by-field", async () => { + const { prisma, createCalls } = makeFakePrisma({ + turns: [{ role: "user", content: "I prefer morning workouts and I'm vegetarian." }], + }); + const runCompletion = vi.fn(async () => + ok( + JSON.stringify([ + { category: "preference", fact: "Prefers morning workouts", confidence: 90 }, + { category: "preference", fact: "Is vegetarian", confidence: 80 }, + ]), + ), + ); + + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + + expect(res).toEqual({ status: "stored", count: 2 }); + expect(createCalls).toHaveLength(2); + // Field-by-field — exact key set, no spread of the parsed object. + for (const data of createCalls) { + expect(Object.keys(data).sort()).toEqual( + ["category", "confidence", "factEncrypted", "sourceConversationId", "userId"].sort(), + ); + expect(data.userId).toBe("user-1"); + expect(data.sourceConversationId).toBe("conv-1"); + expect(data.factEncrypted).toBeInstanceOf(Uint8Array); + } + // Annotation carries counts/ids only. + const extracted = annotateMock.mock.calls.find( + (c) => (c[0] as { action?: { name?: string } }).action?.name === "coach.facts.extracted", + ); + expect(extracted).toBeTruthy(); + expect((extracted![0] as { meta: Record }).meta).toEqual({ + count: 2, + conversationId: "conv-1", + }); + }); + + it("(b) drops items the Zod gate rejects (bad category, over-length)", async () => { + const { prisma, createCalls } = makeFakePrisma({ + turns: [{ role: "user", content: "talk" }], + }); + const tooLong = "x".repeat(FACT_MAX_CHARS + 5); + const runCompletion = vi.fn(async () => + ok( + JSON.stringify([ + { category: "preference", fact: "Prefers tea", confidence: 70 }, // valid + { category: "diagnosis", fact: "Has X", confidence: 90 }, // bad category + { category: "goal", fact: tooLong, confidence: 60 }, // over length + { category: "context", fact: "", confidence: 50 }, // empty + ]), + ), + ); + + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + + expect(res).toEqual({ status: "stored", count: 1 }); + expect(createCalls).toHaveLength(1); + expect(createCalls[0].category).toBe("preference"); + }); + + it("(c) malformed JSON → parse_failed annotation, 0 stored", async () => { + const { prisma, createCalls } = makeFakePrisma({ + turns: [{ role: "user", content: "talk" }], + }); + const runCompletion = vi.fn(async () => ok("not json at all {{{")); + + const res = await extractAndStoreFacts("conv-9", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + + expect(res).toEqual({ status: "none", count: 0 }); + expect(createCalls).toHaveLength(0); + const failed = annotateMock.mock.calls.find( + (c) => (c[0] as { action?: { name?: string } }).action?.name === "coach.facts.parse_failed", + ); + expect(failed).toBeTruthy(); + expect((failed![0] as { meta: Record }).meta).toEqual({ + conversationId: "conv-9", + }); + }); + + it("(d) de-dup drops a near-duplicate of an existing fact", async () => { + const { prisma, createCalls } = makeFakePrisma({ + activeFacts: [{ text: "Prefers morning workouts", confidence: 80 }], + turns: [{ role: "user", content: "talk" }], + }); + const runCompletion = vi.fn(async () => + ok( + JSON.stringify([ + { category: "preference", fact: "prefers morning workouts", confidence: 90 }, // dup + { category: "goal", fact: "Wants to run a 10k in autumn", confidence: 85 }, // new + ]), + ), + ); + + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + + expect(res).toEqual({ status: "stored", count: 1 }); + expect(createCalls).toHaveLength(1); + expect(createCalls[0].category).toBe("goal"); + }); + + it("(e) cap enforcement: at MAX, only strictly-higher-confidence is stored", async () => { + // Fill to the cap; weakest active fact has confidence 30. + const activeFacts = Array.from({ length: MAX_FACTS_PER_USER }, (_, i) => ({ + text: `active fact number ${i}`, + confidence: i === 0 ? 30 : 70, + })); + const { prisma, createCalls } = makeFakePrisma({ + activeFacts, + turns: [{ role: "user", content: "talk" }], + }); + const runCompletion = vi.fn(async () => + ok( + JSON.stringify([ + { category: "goal", fact: "low confidence newcomer", confidence: 20 }, // ≤30 → skip + { category: "goal", fact: "high confidence newcomer", confidence: 95 }, // >30 → store + ]), + ), + ); + + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + + expect(res).toEqual({ status: "stored", count: 1 }); + expect(createCalls).toHaveLength(1); + expect(createCalls[0].confidence).toBe(95); + }); + + it("(f) none/timeout/error provider result → skipped", async () => { + for (const kind of ["none", "timeout", "error"] as const) { + const { prisma, createCalls } = makeFakePrisma({ + turns: [{ role: "user", content: "talk" }], + }); + const runCompletion = vi.fn(async () => ({ kind }) as StatusProviderResult); + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + expect(res).toEqual({ status: "skipped", count: 0 }); + expect(createCalls).toHaveLength(0); + } + }); + + it("returns [] handling: empty array → none, 0 stored", async () => { + const { prisma, createCalls } = makeFakePrisma({ + turns: [{ role: "user", content: "talk" }], + }); + const runCompletion = vi.fn(async () => ok("[]")); + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + expect(res).toEqual({ status: "none", count: 0 }); + expect(createCalls).toHaveLength(0); + }); + + it("skips when the conversation has no decryptable turns", async () => { + const { prisma, createCalls } = makeFakePrisma({ turns: [] }); + const runCompletion = vi.fn(async () => ok("[]")); + const res = await extractAndStoreFacts("conv-1", "user-1", { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + prisma: prisma as any, + // eslint-disable-next-line @typescript-eslint/no-explicit-any + runCompletion: runCompletion as any, + }); + expect(res).toEqual({ status: "skipped", count: 0 }); + expect(runCompletion).not.toHaveBeenCalled(); + expect(createCalls).toHaveLength(0); + }); +}); diff --git a/src/lib/ai/coach/bytes-codec.ts b/src/lib/ai/coach/bytes-codec.ts new file mode 100644 index 000000000..5154a3202 --- /dev/null +++ b/src/lib/ai/coach/bytes-codec.ts @@ -0,0 +1,32 @@ +/** + * Shared AES-256-GCM ↔ `Bytes` codec for the Coach's encrypted columns. + * + * `coach_messages.encrypted_content`, `coach_conversations.summary_encrypted`, + * and `coach_facts.fact_encrypted` all store the same payload shape: the + * `encrypt()` string format from `@/lib/crypto` (`"."`) encoded + * as UTF-8 bytes. Centralised here so there is one ArrayBuffer-backed + * implementation every Coach persistence path shares. + * + * Prisma's `Bytes` type maps to `Uint8Array`, not Node's + * `Buffer`, so we allocate a fresh ArrayBuffer-backed + * `Uint8Array` for writes to keep the structural type stable across Node + * versions. + */ +import { Buffer } from "node:buffer"; + +import { decrypt, encrypt } from "@/lib/crypto"; + +/** Encrypt a UTF-8 string into the `Bytes` payload the schema stores. */ +export function encryptToBytes(plaintext: string): Uint8Array { + const ciphertext = encrypt(plaintext); + const encoded = Buffer.from(ciphertext, "utf8"); + const out = new Uint8Array(new ArrayBuffer(encoded.byteLength)); + out.set(encoded); + return out; +} + +/** Decrypt a `Bytes` payload back to its plaintext. Throws on a bad key id. */ +export function decryptFromBytes(buf: Uint8Array): string { + const text = Buffer.from(buf).toString("utf8"); + return decrypt(text); +} diff --git a/src/lib/ai/coach/coach-memory-refresh-worker.ts b/src/lib/ai/coach/coach-memory-refresh-worker.ts new file mode 100644 index 000000000..92176444b --- /dev/null +++ b/src/lib/ai/coach/coach-memory-refresh-worker.ts @@ -0,0 +1,55 @@ +/** + * v1.11.1 — worker pipeline for the combined Coach memory-refresh queue. + * + * Runs both background generators for one conversation, sequentially so they + * share the wake-up but each pays its own budget check inside + * `runStatusCompletion`: the rolling conversation summary first, then durable + * fact extraction. Each step is fault-isolated — a failure or no-provider in + * one never sinks the other or the job. Kept out of the route bundle (the + * route imports only `enqueueCoachMemoryRefresh` from `coach-memory-shared`). + */ +import { annotate } from "@/lib/logging/context"; + +import type { CoachMemoryRefreshPayload } from "./coach-memory-shared"; +import { extractAndStoreFacts } from "./facts"; +import { refreshConversationSummary } from "./conversation-summary"; + +export async function runCoachMemoryRefresh( + payload: CoachMemoryRefreshPayload, +): Promise { + const { conversationId, userId } = payload; + const locale = payload.locale ?? "de"; + + let summaryStatus = "error"; + try { + const result = await refreshConversationSummary(conversationId, userId, { + locale, + }); + summaryStatus = result.status; + } catch (err) { + annotate({ + action: { name: "coach.memory.refresh.summary_failed" }, + meta: { error: err instanceof Error ? err.message : String(err) }, + }); + } + + let factsStatus = "error"; + let factsCount = 0; + try { + const result = await extractAndStoreFacts(conversationId, userId, { + locale, + }); + factsStatus = result.status; + factsCount = result.count; + } catch (err) { + annotate({ + action: { name: "coach.memory.refresh.facts_failed" }, + meta: { error: err instanceof Error ? err.message : String(err) }, + }); + } + + annotate({ + action: { name: "coach.memory.refresh.done" }, + meta: { summaryStatus, factsStatus, factsCount }, + }); +} diff --git a/src/lib/ai/coach/coach-memory-shared.ts b/src/lib/ai/coach/coach-memory-shared.ts new file mode 100644 index 000000000..31fc00b2e --- /dev/null +++ b/src/lib/ai/coach/coach-memory-shared.ts @@ -0,0 +1,64 @@ +/** + * v1.11.1 — generator-free contract for the Coach long-term-memory refresh + * queue. The chat route enqueues a single-conversation refresh here without + * importing the concrete generators (which would pull the provider chain into + * the route bundle). The worker handler (`runCoachMemoryRefresh`) calls the + * summary + fact generators; it re-uses the queue name from here so there is + * one source of truth. + * + * One combined queue does BOTH conversation-summary refresh and durable-fact + * extraction in a single worker job: when a long conversation crosses the + * history-window cap, both are usually due together, so one job + one + * singleton window avoids a second queue and a redundant wake-up. + * + * Mirrors `period-narrative-shared.ts`: queue name + payload type + enqueue + * helper here, the concrete dispatch in the worker. + */ +import { getGlobalBoss } from "@/lib/jobs/boss-instance"; +import { annotate } from "@/lib/logging/context"; + +export const COACH_MEMORY_REFRESH_QUEUE = "coach-memory-refresh"; + +export interface CoachMemoryRefreshPayload { + conversationId: string; + userId: string; + /** Locale to compose the summary / facts prose in; defaults to "de". */ + locale?: "de" | "en"; +} + +/** + * Fire-and-forget enqueue from the chat turn once a conversation has grown past + * the history-window cap. A `singletonKey` per conversation collapses repeated + * turns within a short window into one queued job. No-ops cleanly when the + * global boss is unavailable (a web process without an embedded worker) — the + * memory simply stays as-is until the next eligible turn. + */ +export async function enqueueCoachMemoryRefresh(payload: { + conversationId: string; + userId: string; + locale: "de" | "en"; +}): Promise { + const boss = getGlobalBoss(); + if (!boss) return; + try { + await boss.send( + COACH_MEMORY_REFRESH_QUEUE, + { + conversationId: payload.conversationId, + userId: payload.userId, + locale: payload.locale, + } satisfies CoachMemoryRefreshPayload, + { + singletonKey: `refresh:${payload.conversationId}`, + singletonSeconds: 120, + }, + ); + annotate({ + action: { name: "coach.memory.refresh.enqueued" }, + meta: { locale: payload.locale }, + }); + } catch { + // Best-effort — a failure just leaves the Coach memory unchanged until the + // next eligible turn re-enqueues. + } +} diff --git a/src/lib/ai/coach/conversation-summary.ts b/src/lib/ai/coach/conversation-summary.ts new file mode 100644 index 000000000..06e28c4a8 --- /dev/null +++ b/src/lib/ai/coach/conversation-summary.ts @@ -0,0 +1,234 @@ +/** + * v1.11.1 (Epic B, B-W5b) — Coach conversation-summary COMPUTE module. + * + * A rolling, encrypted natural-language summary of the turns BEFORE the most + * recent `RECENT_HISTORY` window. When a conversation grows past `TURN_CAP` + * the chat route folds the older turns into a single synthetic line; today + * that line is a dead placeholder. This module produces a durable summary the + * route can substitute for the placeholder so the Coach keeps memory of the + * elided turns. + * + * Structural sibling of `period-narrative-generate.ts`: + * - freshness short-circuit before any provider call, + * - `runStatusCompletion` over the user's provider chain (NOT hand-rolled), + * - `encryptToBytes` → `update` of the single conversation row, + * - `annotate` carries counts + ids only, never the summary content. + * + * Background generation is intentionally UNBILLED — like the period-narrative + * warm, it does not call `recordSpend`. The operator's per-user daily token + * ceiling is the cost backstop; the interactive reply meter is not charged for + * background memory upkeep. + * + * Fail-closed crypto: an undecryptable prior summary is treated as absent + * (regenerate from scratch) and never allowed to throw into the caller. + */ +import type { PrismaClient } from "@/generated/prisma/client"; +import { prisma as defaultPrisma } from "@/lib/db"; +import { annotate } from "@/lib/logging/context"; +import { + runStatusCompletion, + type StatusProviderResult, +} from "@/lib/insights/status-provider"; + +import { decryptFromBytes, encryptToBytes } from "./bytes-codec"; + +/** + * Safety cap on the stored summary length (~150 tokens). The prompt asks for + * a short paragraph; this backstops a misbehaving provider so the encrypted + * column + the next prompt's history window stay bounded. + */ +const SUMMARY_TARGET_CHARS = 600; + +/** Re-summarise only once at least this many new turns accumulate. */ +export const SUMMARY_REFRESH_TURN_DELTA = 6; + +/** + * Mirror of the chat route's history-window constants (`route.ts:75-76`). + * Defined locally — the route is a 600+ LOC server module we must not import + * from here. Keep in sync with the route by hand if either changes. + */ +const TURN_CAP = 20; +const RECENT_HISTORY = 18; + +/** Injected provider-completion shape (defaults to the real chain runner). */ +type RunCompletion = typeof runStatusCompletion; + +export interface RefreshConversationSummaryOptions { + now?: Date; + runCompletion?: RunCompletion; + prisma?: Pick; + locale?: "de" | "en"; +} + +export type RefreshConversationSummaryResult = { + status: "fresh" | "generated" | "skipped" | "insufficient"; +}; + +/** A conversation turn folded into the summary (role-prefixed in the prompt). */ +export interface SummaryFoldTurn { + role: "user" | "assistant"; + content: string; +} + +const SYSTEM_PROMPT_EN = `You compress a coaching conversation into a durable rolling summary for the assistant's own future memory. Write 2-4 sentences (<= ~120 words) of plain prose. Capture: what the user is working on or worried about, any goals or preferences they stated, decisions or agreements reached, and open threads to follow up. Be descriptive, never diagnostic — record what was said, do not infer conditions. EXCLUDE: one-off pleasantries, exact numbers (those live in the live snapshot), anything the user asked you to forget, and any detail not useful to continuing the conversation. If a PRIOR SUMMARY is supplied, MERGE the new turns into it and return the merged summary — do not append; rewrite the whole thing concisely. Output only the summary prose, no preamble.`; + +const SYSTEM_PROMPT_DE = `Du verdichtest ein Coaching-Gespräch zu einer dauerhaften, fortlaufenden Zusammenfassung für das eigene künftige Gedächtnis des Assistenten. Schreibe 2-4 Sätze (<= ~120 Wörter) in einfacher Prosa. Erfasse: woran die Person arbeitet oder was sie beschäftigt, genannte Ziele oder Vorlieben, getroffene Entscheidungen oder Vereinbarungen und offene Punkte zum Nachfassen. Sei beschreibend, nie diagnostisch — halte fest, was gesagt wurde, leite keine Erkrankungen ab. AUSSCHLIESSEN: einmalige Höflichkeiten, exakte Zahlen (die stehen im Live-Snapshot), alles, worum die Person dich zu vergessen gebeten hat, und jedes Detail, das für die Fortsetzung des Gesprächs nicht nützlich ist. Wenn eine FRÜHERE ZUSAMMENFASSUNG vorliegt, FÜGE die neuen Wortwechsel in sie EIN und gib die zusammengeführte Fassung zurück — nicht anhängen; schreibe das Ganze knapp neu. Gib nur die Zusammenfassungsprosa aus, ohne Vorrede.`; + +/** + * Build the user prompt fed to the model from the prior summary (when any) and + * the turns being folded in. Exported so tests can assert the merge path + * carries the prior summary into the prompt. + */ +export function buildSummaryUserPrompt( + priorSummary: string | null, + foldedTurns: SummaryFoldTurn[], + locale: "de" | "en" = "en", +): string { + const priorLabel = locale === "de" ? "FRÜHERE ZUSAMMENFASSUNG" : "PRIOR SUMMARY"; + const turnsLabel = locale === "de" ? "NEUE WORTWECHSEL" : "NEW TURNS"; + const none = locale === "de" ? "(keine)" : "(none)"; + + const priorBlock = priorSummary && priorSummary.trim().length > 0 + ? priorSummary.trim() + : none; + + const turnsBlock = foldedTurns + .map((t) => `${t.role}: ${t.content}`) + .join("\n"); + + return `${priorLabel}\n${priorBlock}\n\n${turnsLabel}\n${turnsBlock}`; +} + +/** + * Decrypt the prior summary fail-closed: a missing or undecryptable row yields + * `null` so the caller regenerates from scratch rather than throwing into a + * background job (mirrors `readPeriodNarrative`'s undecryptable-as-absent rule). + */ +function decryptPriorSummary(buf: Uint8Array | null): string | null { + if (!buf || buf.byteLength === 0) return null; + try { + return decryptFromBytes(buf); + } catch { + return null; + } +} + +/** + * (Re)compute the rolling summary for one conversation. + * + * Ownership-scoped: a conversation not owned by `userId` is treated as absent + * (`"insufficient"`). Best-effort: a no-provider / timeout / error completion + * leaves any existing summary untouched and returns `"skipped"`. + */ +export async function refreshConversationSummary( + conversationId: string, + userId: string, + opts?: RefreshConversationSummaryOptions, +): Promise { + const now = opts?.now ?? new Date(); + const prisma = opts?.prisma ?? defaultPrisma; + const runCompletion = opts?.runCompletion ?? runStatusCompletion; + const locale = opts?.locale ?? "en"; + + // 1. Ownership-scoped load. Decrypt every message body for the fold slice. + const conversation = await prisma.coachConversation.findFirst({ + where: { id: conversationId, userId }, + select: { + id: true, + summaryEncrypted: true, + summaryTurnCount: true, + messages: { + orderBy: { createdAt: "asc" }, + select: { role: true, encryptedContent: true }, + }, + }, + }); + if (!conversation) { + return { status: "insufficient" }; + } + + const turns = conversation.messages; + const turnCount = turns.length; + + // 2. Cheap short-circuit: nothing has been elided yet — the live window + // already carries the whole conversation. + if (turnCount <= TURN_CAP) { + return { status: "insufficient" }; + } + + // The high-water mark of turns this refresh would fold: everything before + // the most-recent `RECENT_HISTORY` window (same slice the route elides). + const foldHighWater = turnCount - RECENT_HISTORY; + + // 3. Don't pay a provider call for a couple of new turns. + if ( + conversation.summaryTurnCount > 0 && + foldHighWater - conversation.summaryTurnCount < SUMMARY_REFRESH_TURN_DELTA + ) { + return { status: "fresh" }; + } + + // 4. Decrypt prior summary (fail-closed) + the older turns to fold. + const priorSummary = decryptPriorSummary(conversation.summaryEncrypted); + const foldedTurns: SummaryFoldTurn[] = turns + .slice(0, foldHighWater) + .flatMap((m) => { + let content: string; + try { + content = decryptFromBytes(m.encryptedContent); + } catch { + // Skip an undecryptable message (e.g. a row mid key-rotation) instead + // of throwing and permanently stalling this conversation's summary — + // fail-closed like every other decrypt in the Coach stack. + return []; + } + return [ + { role: m.role === "assistant" ? "assistant" : "user", content }, + ]; + }); + + // 5. Run the user's provider chain. Best-effort: non-ok leaves the old + // summary in place. + const completion: StatusProviderResult = await runCompletion({ + userId, + cacheAction: "coach.summary", + systemPrompt: locale === "de" ? SYSTEM_PROMPT_DE : SYSTEM_PROMPT_EN, + userPrompt: buildSummaryUserPrompt(priorSummary, foldedTurns, locale), + temperature: 0.3, + maxTokens: 200, + }); + + if (completion.kind !== "ok") { + return { status: "skipped" }; + } + + const trimmed = completion.content.trim(); + if (trimmed.length === 0) { + return { status: "skipped" }; + } + // Backstop the stored length even if the provider over-ran the prompt's + // "short paragraph" instruction. + const text = + trimmed.length > SUMMARY_TARGET_CHARS + ? trimmed.slice(0, SUMMARY_TARGET_CHARS) + : trimmed; + + // 6. Encrypt + persist. Field-by-field data object (no spread). + const summaryEncrypted = encryptToBytes(text); + await prisma.coachConversation.update({ + where: { id: conversationId }, + data: { + summaryEncrypted, + summaryUpdatedAt: now, + summaryTurnCount: foldHighWater, + }, + }); + + // 7. Counts + ids only — never the summary text. + annotate({ + action: { name: "coach.summary.generated" }, + meta: { foldedTurns: foldHighWater, conversationId }, + }); + + return { status: "generated" }; +} diff --git a/src/lib/ai/coach/facts.ts b/src/lib/ai/coach/facts.ts new file mode 100644 index 000000000..52c1d0b4e --- /dev/null +++ b/src/lib/ai/coach/facts.ts @@ -0,0 +1,421 @@ +/** + * v1.11.1 (Epic B, B-W7) — durable personal-fact extraction + injection. + * + * The Coach learns STABLE facts about a user from a conversation — standing + * preferences, conditions the user STATES about themselves, durable goals, + * standing constraints, durable life context — and persists them encrypted + * at rest so future turns can personalise without re-asking. Facts are + * DESCRIPTIVE (the user's own framing), never diagnostic: the extractor is + * forbidden from inferring a medical condition or recording a measurement. + * + * Two halves: + * - `extractAndStoreFacts` — the background compute: load active facts + + * recent turns, run one bounded `runStatusCompletion`, defensively parse + * the JSON array, drop anything the Zod gate rejects, de-dup against the + * active set, enforce a per-user cap, and persist survivors field-by-field + * (no mass-assignment spread). + * - `buildCoachFactsBlock` — the injection read: top-N active facts ranked + * by confidence then recency, decrypted fault-isolated (an undecryptable + * row is skipped, never thrown), for the snapshot memory block. + * + * The fact TEXT is encrypted via `bytes-codec.ts` (the same AES-256-GCM codec + * as `CoachMessage`). category/confidence/sourceConversationId stay plain so + * the ranker can sort without paying a per-row decrypt. Annotations carry + * counts + ids only — fact text is NEVER logged. + * + * Server-only — reads `@/lib/db`. + */ +import { z } from "zod"; + +import { prisma } from "@/lib/db"; +import { runStatusCompletion } from "@/lib/insights/status-provider"; +import { annotate } from "@/lib/logging/context"; + +import { decryptFromBytes, encryptToBytes } from "./bytes-codec"; + +/** Closed category enum — app-side (no DB enum), matching the schema column. */ +export const COACH_FACT_CATEGORIES = [ + "preference", + "condition", + "goal", + "constraint", + "context", +] as const; + +export type CoachFactCategory = (typeof COACH_FACT_CATEGORIES)[number]; + +/** Hard cap on active facts per user; at cap, only strictly-higher-confidence facts displace. */ +export const MAX_FACTS_PER_USER = 50; +/** Per-fact text length cap (mirrors the Zod gate + the prompt instruction). */ +export const FACT_MAX_CHARS = 160; +/** How many active facts the injection block carries into the snapshot. */ +export const FACTS_INJECT_TOP_N = 8; + +/** Cap on recent turns fed into the extraction prompt (bounds prompt size). */ +const RECENT_TURNS_CAP = 10; + +type RunCompletionFn = typeof runStatusCompletion; +type PrismaLike = Pick; + +interface ExtractOpts { + now?: Date; + runCompletion?: RunCompletionFn; + prisma?: PrismaLike; + locale?: string; +} + +interface BuildBlockOpts { + prisma?: PrismaLike; +} + +// --------------------------------------------------------------------------- +// Extraction prompt (EN + DE mirror) +// --------------------------------------------------------------------------- + +const EXTRACTION_PROMPT_EN = `You extract DURABLE personal facts about the user from a coaching conversation, for the assistant's long-term memory. Return a JSON array (no prose, no fences). Each item: { "category": one of preference|condition|goal|constraint|context, "fact": "", "confidence": 0-100 }. +ONLY extract facts that are STABLE and will still be true next month: standing preferences, conditions the user STATES about themselves, durable goals, standing constraints, durable life context. Record descriptively in the user's own framing — NEVER diagnose, infer a medical condition the user did not state, or record a number/measurement (those live in the data). EXCLUDE: transient feelings ("tired today"), one-off events, anything time-bound, sensitive detail beyond what is needed to coach, and anything the user asked you to forget. If nothing durable was said, return []. Prefer FEW high-confidence facts over many speculative ones.`; + +const EXTRACTION_PROMPT_DE = `Du extrahierst DAUERHAFTE persönliche Fakten über die Nutzerin oder den Nutzer aus einem Coaching-Gespräch für das Langzeitgedächtnis des Assistenten. Gib ein JSON-Array zurück (kein Fließtext, keine Code-Zäune). Jedes Element: { "category": eines von preference|condition|goal|constraint|context, "fact": "", "confidence": 0-100 }. +Extrahiere NUR Fakten, die STABIL sind und auch nächsten Monat noch zutreffen: dauerhafte Vorlieben, Bedingungen, die die Person SELBST über sich AUSSAGT, dauerhafte Ziele, dauerhafte Einschränkungen, dauerhafter Lebenskontext. Halte sie beschreibend in der eigenen Formulierung der Person fest — DIAGNOSTIZIERE NIEMALS, leite keine medizinische Bedingung ab, die die Person nicht selbst genannt hat, und erfasse keine Zahl oder Messung (die liegen in den Daten). SCHLIESSE AUS: vorübergehende Gefühle ("heute müde"), einmalige Ereignisse, alles Zeitgebundene, sensible Details über das fürs Coaching Nötige hinaus und alles, worum die Person gebeten hat, es zu vergessen. Wenn nichts Dauerhaftes gesagt wurde, gib [] zurück. Bevorzuge WENIGE Fakten mit hoher Sicherheit gegenüber vielen spekulativen.`; + +function extractionSystemPrompt(locale: string | undefined): string { + return locale?.toLowerCase().startsWith("de") + ? EXTRACTION_PROMPT_DE + : EXTRACTION_PROMPT_EN; +} + +// --------------------------------------------------------------------------- +// Zod gate for the model's JSON array +// --------------------------------------------------------------------------- + +const rawFactSchema = z.object({ + category: z.enum(COACH_FACT_CATEGORIES), + fact: z + .string() + .trim() + .min(1) + .max(FACT_MAX_CHARS), + confidence: z.coerce.number().int().min(0).max(100), +}); + +const rawFactArraySchema = z.array(z.unknown()); + +interface ParsedFact { + category: CoachFactCategory; + fact: string; + confidence: number; +} + +/** + * Parse the model output into validated facts. Returns `null` only when the + * top-level JSON is unparseable (the caller then annotates `parse_failed`); + * individual malformed items are dropped silently, not fatal. + */ +function parseFacts(content: string): { facts: ParsedFact[]; dropped: number } | null { + let json: unknown; + try { + json = JSON.parse(content.trim()); + } catch { + return null; + } + + const arr = rawFactArraySchema.safeParse(json); + if (!arr.success) return null; + + const facts: ParsedFact[] = []; + let dropped = 0; + for (const item of arr.data) { + const parsed = rawFactSchema.safeParse(item); + if (parsed.success) { + facts.push({ + category: parsed.data.category, + fact: parsed.data.fact, + confidence: Math.max(0, Math.min(100, Math.round(parsed.data.confidence))), + }); + } else { + dropped += 1; + } + } + return { facts, dropped }; +} + +// --------------------------------------------------------------------------- +// De-dup — lowercase-normalise + token-overlap against the active set +// --------------------------------------------------------------------------- + +const TOKEN_SPLIT = /[^a-z0-9äöüß]+/i; + +function tokenSet(text: string): Set { + return new Set( + text + .toLowerCase() + .split(TOKEN_SPLIT) + .filter((t) => t.length > 1), + ); +} + +/** Jaccard-style overlap of the two token sets, 0..1. */ +function tokenOverlap(a: Set, b: Set): number { + if (a.size === 0 || b.size === 0) return 0; + let inter = 0; + for (const t of a) if (b.has(t)) inter += 1; + const union = a.size + b.size - inter; + return union === 0 ? 0 : inter / union; +} + +const DEDUP_OVERLAP_THRESHOLD = 0.6; + +function isNearDuplicate(candidate: string, existing: string[]): boolean { + const cTokens = tokenSet(candidate); + const cNorm = candidate.trim().toLowerCase(); + for (const ex of existing) { + if (ex.trim().toLowerCase() === cNorm) return true; + if (tokenOverlap(cTokens, tokenSet(ex)) >= DEDUP_OVERLAP_THRESHOLD) return true; + } + return false; +} + +// --------------------------------------------------------------------------- +// Conversation-turn loading +// --------------------------------------------------------------------------- + +function decryptOrNull(buf: Uint8Array): string | null { + try { + return decryptFromBytes(buf); + } catch { + return null; + } +} + +interface ActiveFactRow { + id: string; + factEncrypted: Uint8Array; + category: string; + confidence: number; +} + +/** Load active facts and decrypt them fault-isolated (undecryptable rows skipped). */ +async function loadActiveFacts( + db: PrismaLike, + userId: string, +): Promise> { + const rows = (await db.coachFact.findMany({ + where: { userId, deletedAt: null }, + select: { id: true, factEncrypted: true, category: true, confidence: true }, + })) as ActiveFactRow[]; + + const out: Array<{ id: string; text: string; category: string; confidence: number }> = []; + for (const r of rows) { + const text = decryptOrNull(r.factEncrypted); + if (text === null) continue; + out.push({ id: r.id, text, category: r.category, confidence: r.confidence }); + } + return out; +} + +interface ConversationTurnRow { + role: string; + encryptedContent: Uint8Array; +} + +/** Load the last `RECENT_TURNS_CAP` turns of a conversation, decrypted. */ +async function loadRecentTurns( + db: PrismaLike, + conversationId: string, + userId: string, +): Promise> { + const row = (await db.coachConversation.findFirst({ + where: { id: conversationId, userId }, + select: { + messages: { + orderBy: { createdAt: "desc" }, + take: RECENT_TURNS_CAP, + select: { role: true, encryptedContent: true }, + }, + }, + })) as { messages: ConversationTurnRow[] } | null; + + if (!row) return []; + + // Loaded newest-first; reverse to chronological for the prompt. + const turns: Array<{ role: string; content: string }> = []; + for (const m of [...row.messages].reverse()) { + const content = decryptOrNull(m.encryptedContent); + if (content === null) continue; + turns.push({ role: m.role, content }); + } + return turns; +} + +function buildUserPrompt( + turns: Array<{ role: string; content: string }>, + activeFacts: string[], +): string { + const transcript = turns.map((t) => `${t.role}: ${t.content}`).join("\n"); + const existing = + activeFacts.length > 0 + ? `\n\nEXISTING FACTS (do not re-emit these):\n${activeFacts.map((f) => `- ${f}`).join("\n")}` + : "\n\nEXISTING FACTS: none"; + return `${transcript}${existing}`; +} + +// --------------------------------------------------------------------------- +// Extraction entry point +// --------------------------------------------------------------------------- + +/** + * Extract durable facts from a conversation and persist the survivors. + * + * Returns: + * - `{status:"stored", count}` — at least one new fact persisted. + * - `{status:"none", count:0}` — the model produced nothing durable (or the + * JSON was unparseable, in which case `coach.facts.parse_failed` is annotated). + * - `{status:"skipped", count:0}` — no usable provider / timeout / error, or + * every candidate was a dup / dropped / displaced by the cap. + */ +export async function extractAndStoreFacts( + conversationId: string, + userId: string, + opts?: ExtractOpts, +): Promise<{ status: "stored" | "none" | "skipped"; count: number }> { + const db = opts?.prisma ?? prisma; + const runCompletion = opts?.runCompletion ?? runStatusCompletion; + + const [active, turns] = await Promise.all([ + loadActiveFacts(db, userId), + loadRecentTurns(db, conversationId, userId), + ]); + + if (turns.length === 0) { + return { status: "skipped", count: 0 }; + } + + const activeTexts = active.map((f) => f.text); + const result = await runCompletion({ + userId, + cacheAction: "coach.facts", + systemPrompt: extractionSystemPrompt(opts?.locale), + userPrompt: buildUserPrompt(turns, activeTexts), + temperature: 0.2, + maxTokens: 300, + }); + + if (result.kind !== "ok") { + return { status: "skipped", count: 0 }; + } + + const parsed = parseFacts(result.content); + if (parsed === null) { + annotate({ + action: { name: "coach.facts.parse_failed" }, + meta: { conversationId }, + }); + return { status: "none", count: 0 }; + } + + if (parsed.facts.length === 0) { + return { status: "none", count: 0 }; + } + + // De-dup against the active set; grow the seen-set as we accept candidates + // so two near-identical candidates in one batch can't both land. + const seen = [...activeTexts]; + const accepted: ParsedFact[] = []; + for (const cand of parsed.facts) { + if (isNearDuplicate(cand.fact, seen)) continue; + accepted.push(cand); + seen.push(cand.fact); + } + + if (accepted.length === 0) { + return { status: "skipped", count: 0 }; + } + + // Cap enforcement: at/over cap, a candidate may only be stored if it is + // strictly higher-confidence than the current lowest-confidence active fact + // (the new row simply outranks it for injection; soft-delete/eviction of the + // displaced row is the management surface's job, not the extractor's). + let lowestActiveConfidence = + active.length > 0 ? Math.min(...active.map((f) => f.confidence)) : -1; + let remainingCapacity = MAX_FACTS_PER_USER - active.length; + + const toStore: ParsedFact[] = []; + // Highest-confidence candidates first so a tight cap admits the best. + for (const cand of [...accepted].sort((a, b) => b.confidence - a.confidence)) { + if (remainingCapacity > 0) { + toStore.push(cand); + remainingCapacity -= 1; + continue; + } + // At cap — only admit if strictly better than the weakest active fact. + if (cand.confidence > lowestActiveConfidence) { + toStore.push(cand); + lowestActiveConfidence = cand.confidence; + } + } + + if (toStore.length === 0) { + return { status: "skipped", count: 0 }; + } + + for (const f of toStore) { + // Field-by-field, no spread (mass-assignment rule, CLAUDE.md). + await db.coachFact.create({ + data: { + userId, + factEncrypted: encryptToBytes(f.fact), + category: f.category, + confidence: f.confidence, + sourceConversationId: conversationId, + }, + }); + } + + annotate({ + action: { name: "coach.facts.extracted" }, + meta: { count: toStore.length, conversationId }, + }); + + return { status: "stored", count: toStore.length }; +} + +// --------------------------------------------------------------------------- +// Injection block +// --------------------------------------------------------------------------- + +interface RankFactRow { + factEncrypted: Uint8Array; + category: string; + confidence: number; + updatedAt: Date; +} + +/** + * Build the top-N active facts for the snapshot memory block, ranked by + * `confidence DESC, updatedAt DESC`. Decrypt is fault-isolated — an + * undecryptable row is skipped, never thrown into the caller. Returns `null` + * when the user has no usable active facts. + */ +export async function buildCoachFactsBlock( + userId: string, + opts?: BuildBlockOpts, +): Promise<{ facts: Array<{ category: string; text: string }> } | null> { + const db = opts?.prisma ?? prisma; + + const rows = (await db.coachFact.findMany({ + where: { userId, deletedAt: null }, + orderBy: [{ confidence: "desc" }, { updatedAt: "desc" }], + select: { factEncrypted: true, category: true, confidence: true, updatedAt: true }, + })) as RankFactRow[]; + + const facts: Array<{ category: string; text: string }> = []; + for (const r of rows) { + if (facts.length >= FACTS_INJECT_TOP_N) break; + const text = decryptOrNull(r.factEncrypted); + if (text === null) continue; + facts.push({ category: r.category, text }); + } + + if (facts.length === 0) return null; + return { facts }; +} diff --git a/src/lib/ai/coach/memory-snapshot.ts b/src/lib/ai/coach/memory-snapshot.ts index ff7371003..36c0a1560 100644 --- a/src/lib/ai/coach/memory-snapshot.ts +++ b/src/lib/ai/coach/memory-snapshot.ts @@ -38,6 +38,7 @@ import { type BandTransition, } from "@/lib/insights/narrative/period-narrative"; import type { BaselineProfile } from "@/lib/insights/derived"; +import { buildCoachFactsBlock } from "./facts"; /** The period the rolling profile recalls — month is the high-signal beat. */ const MEMORY_PERIOD: NarrativePeriod = "month"; @@ -72,6 +73,11 @@ export interface PriorNarrativeRecall { export interface CoachMemoryBlock { priorNarrative?: PriorNarrativeRecall; trendMemory: Record; + /** + * v1.11.1 — durable personal facts the Coach has learned (top-N, ranked by + * confidence then recency). Descriptive, never diagnostic. Absent when none. + */ + facts?: Array<{ category: string; text: string }>; } /** Pull the headline + driver recall off the latest period narrative. */ @@ -147,11 +153,29 @@ export async function buildCoachMemoryBlock( // A context failure leaves trendMemory empty — never sinks the turn. } - if (!priorNarrative && Object.keys(trendMemory).length === 0) { + // Sub-source 3 (v1.11.1): durable personal facts the Coach has extracted. + // Fault-isolated like the others — a read/decrypt failure drops the facts + // sub-block and never sinks the turn. + let facts: Array<{ category: string; text: string }> | undefined; + try { + const factsBlock = await buildCoachFactsBlock(userId); + if (factsBlock && factsBlock.facts.length > 0) { + facts = factsBlock.facts; + } + } catch { + facts = undefined; + } + + if ( + !priorNarrative && + Object.keys(trendMemory).length === 0 && + !facts + ) { return null; } const block: CoachMemoryBlock = { trendMemory }; if (priorNarrative) block.priorNarrative = priorNarrative; + if (facts) block.facts = facts; return block; } diff --git a/src/lib/ai/coach/persistence.ts b/src/lib/ai/coach/persistence.ts index 22a6141e5..9d0c0440c 100644 --- a/src/lib/ai/coach/persistence.ts +++ b/src/lib/ai/coach/persistence.ts @@ -10,10 +10,8 @@ * provenance (window names, metric tags, sample counts) and never raw * values, so it can be queried without decryption for analytics. */ -import { Buffer } from "node:buffer"; - import { prisma } from "@/lib/db"; -import { encrypt, decrypt } from "@/lib/crypto"; +import { decryptFromBytes, encryptToBytes } from "./bytes-codec"; import type { CoachConversationDTO, @@ -49,27 +47,6 @@ export function summariseTitle(input: string): string { return `${cut.trimEnd()}…`; } -/** - * Encode a UTF-8 string as the AES-256-GCM payload format the schema - * stores in `coach_messages.encrypted_content`. - * - * Prisma's `Bytes` type maps to `Uint8Array`, not Node's - * `Buffer`. We allocate a fresh ArrayBuffer-backed - * Uint8Array so the structural type matches across Node versions. - */ -function encryptToBytes(plaintext: string): Uint8Array { - const ciphertext = encrypt(plaintext); - const encoded = Buffer.from(ciphertext, "utf8"); - const out = new Uint8Array(new ArrayBuffer(encoded.byteLength)); - out.set(encoded); - return out; -} - -function decryptFromBytes(buf: Uint8Array): string { - const text = Buffer.from(buf).toString("utf8"); - return decrypt(text); -} - function provenanceToJson(provenance: CoachProvenance | null): string | null { if (!provenance) return null; return JSON.stringify(provenance); @@ -241,6 +218,18 @@ export async function fetchConversationWithMessages( promptVersion: m.promptVersion, })); + // v1.11.1 — decrypt the rolling conversation summary (fail-closed: an + // undecryptable row is treated as absent so the chat turn never throws and + // simply falls back to the placeholder). + let summary: string | null = null; + if (row.summaryEncrypted && row.summaryEncrypted.byteLength > 0) { + try { + summary = decryptFromBytes(row.summaryEncrypted); + } catch { + summary = null; + } + } + return { id: row.id, title: row.title, @@ -248,6 +237,7 @@ export async function fetchConversationWithMessages( updatedAt: row.updatedAt.toISOString(), messageCount: messages.length, messages, + summary, }; } diff --git a/src/lib/ai/coach/system-prompt.ts b/src/lib/ai/coach/system-prompt.ts index ac2a384a4..9ae0d0542 100644 --- a/src/lib/ai/coach/system-prompt.ts +++ b/src/lib/ai/coach/system-prompt.ts @@ -200,6 +200,14 @@ ISO-week means. not clinical assessments or diagnoses — never frame a band as a medical finding, and lean on lower-confidence / few-day entries cautiously. Never recompute or second-guess the number. +- The SNAPSHOT's "memory" block MAY carry a "facts" list — durable + things you have learned about this user across conversations (stable + preferences, conditions they have told you about, goals, constraints, + life context). Use them to personalise your reply and to avoid + re-asking what you already know. Treat each as the user's OWN stated + context, DESCRIPTIVE not diagnostic — never restate a "condition" fact + as a medical finding, and never invent a fact the block does not + carry. If a fact seems outdated, gently check it rather than assume. EVIDENCE BLOCK @@ -471,6 +479,16 @@ ISO-Wochenmittel zusammen. das der SNAPSHOT nicht enthält. Der "workouts"-Block führt die jüngsten Einheiten plus eine Zusammenfassung je Sportart, nicht jede Einheit. +- Der "memory"-Block des SNAPSHOT kann eine "facts"-Liste tragen — + dauerhafte Dinge, die du über diesen Nutzer gelernt hast (stabile + Vorlieben, vom Nutzer selbst genannte gesundheitliche Umstände, + Ziele, Einschränkungen, Lebenskontext). Nutze sie, um deine Antwort + zu personalisieren und nicht erneut zu fragen, was du schon weißt. + Behandle jeden Eintrag als die EIGENE Aussage des Nutzers, + BESCHREIBEND, nicht diagnostisch — formuliere einen "condition"-Fakt + nie als medizinischen Befund um und erfinde nie einen Fakt, den der + Block nicht enthält. Wirkt ein Fakt veraltet, frage behutsam nach, + statt es anzunehmen. EVIDENZ-BLOCK diff --git a/src/lib/ai/coach/types.ts b/src/lib/ai/coach/types.ts index e676c1c33..e5dcda51d 100644 --- a/src/lib/ai/coach/types.ts +++ b/src/lib/ai/coach/types.ts @@ -299,6 +299,11 @@ export interface CoachMessageDTO { export interface CoachConversationDetailDTO extends CoachConversationDTO { messages: CoachMessageDTO[]; + /** + * v1.11.1 — decrypted rolling summary of the turns elided past the history + * window, or null when none is on file / it could not be decrypted. + */ + summary?: string | null; } /** diff --git a/src/lib/analytics/__tests__/bp-in-target-fast-path.test.ts b/src/lib/analytics/__tests__/bp-in-target-fast-path.test.ts index 5e781df45..871bbaa32 100644 --- a/src/lib/analytics/__tests__/bp-in-target-fast-path.test.ts +++ b/src/lib/analytics/__tests__/bp-in-target-fast-path.test.ts @@ -12,6 +12,9 @@ vi.mock("@/lib/db", () => ({ prisma: { measurement: { findMany: vi.fn() }, measurementRollup: { findMany: vi.fn() }, + // v1.11.1 — readRollupBuckets lazy-loads the source-priority blob; null + // (default findUnique) falls back to the default ladders. + user: { findUnique: vi.fn() }, }, })); diff --git a/src/lib/analytics/__tests__/correlations-fast-path.test.ts b/src/lib/analytics/__tests__/correlations-fast-path.test.ts index d8acd1865..6add0b368 100644 --- a/src/lib/analytics/__tests__/correlations-fast-path.test.ts +++ b/src/lib/analytics/__tests__/correlations-fast-path.test.ts @@ -13,6 +13,8 @@ vi.mock("@/lib/db", () => ({ prisma: { measurement: { findMany: vi.fn() }, measurementRollup: { findMany: vi.fn() }, + // v1.11.1 — readRollupBuckets lazy-loads the source-priority blob. + user: { findUnique: vi.fn() }, moodEntry: { findMany: vi.fn() }, medicationIntakeEvent: { findMany: vi.fn() }, }, diff --git a/src/lib/analytics/__tests__/health-score-fast-path.test.ts b/src/lib/analytics/__tests__/health-score-fast-path.test.ts index d7f645e6d..3f5d47a22 100644 --- a/src/lib/analytics/__tests__/health-score-fast-path.test.ts +++ b/src/lib/analytics/__tests__/health-score-fast-path.test.ts @@ -12,6 +12,8 @@ vi.mock("@/lib/db", () => ({ prisma: { measurement: { findMany: vi.fn() }, measurementRollup: { findMany: vi.fn() }, + // v1.11.1 — readRollupBuckets lazy-loads the source-priority blob. + user: { findUnique: vi.fn() }, moodEntry: { findMany: vi.fn() }, medication: { findMany: vi.fn() }, medicationIntakeEvent: { findMany: vi.fn() }, diff --git a/src/lib/analytics/__tests__/summaries-slice.test.ts b/src/lib/analytics/__tests__/summaries-slice.test.ts index 53d1b2286..7dcee10a7 100644 --- a/src/lib/analytics/__tests__/summaries-slice.test.ts +++ b/src/lib/analytics/__tests__/summaries-slice.test.ts @@ -10,6 +10,16 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; vi.mock("@/lib/db", () => ({ prisma: { $queryRaw: vi.fn(), + // v1.11.1 — the data-aggregate queries inside `computeFromRollups` + // and `computeFromLiveAggregate` now splice a whitelisted + // source-rank CASE and bind `userId` as `$1`, so they run via + // `$queryRawUnsafe(sql, userId)` rather than the tagged-template + // `$queryRaw`. The coverage probe stays on `$queryRaw`. + $queryRawUnsafe: vi.fn(), + // v1.11.1 — `loadUserSourcePriority` reads the user's + // `sourcePriorityJson` to build the rank ladders. `null` here → + // default ladders. + user: { findUnique: vi.fn() }, measurement: { findFirst: vi.fn() }, // v1.4.36 — slim slice reads DAY buckets from `measurement_rollups` // on the happy path. The freshness watermark inside @@ -26,22 +36,63 @@ vi.mock("@/lib/logging/context", () => ({ annotate: vi.fn(), })); +// v1.11.1 — `computeFromRollups` / `computeFromLiveAggregate` build a +// source-rank CASE via `@/lib/analytics/source-rank-sql` and splice it +// into the data-aggregate SQL. The builder's own correctness is pinned +// in its dedicated suite (and the integration suite runs the real SQL); +// here we stub it to deterministic, side-effect-free fragments so the +// slice's plumbing — path selection, slope/round/empty contracts, the +// 90-day FILTER caps the slice itself writes — is exercised without +// coupling the slice unit test to the rank builder's enum-whitelist +// internals. +vi.mock("@/lib/analytics/source-rank-sql", () => ({ + buildSourceRankCase: vi.fn(() => "90"), + canonicalMeasurementsFrom: vi.fn( + (_rank: string, sinceInterval?: string) => + `( + SELECT mm.* + FROM measurements mm + WHERE mm."user_id" = $1 + AND mm."deleted_at" IS NULL + ${ + sinceInterval + ? `AND mm."measured_at" >= NOW() - INTERVAL '${sinceInterval}'` + : "" + } + ) m`, + ), +})); + import { prisma } from "@/lib/db"; +import { canonicalMeasurementsFrom } from "@/lib/analytics/source-rank-sql"; import { computeSummariesSlice } from "../summaries-slice"; const RAW = prisma.$queryRaw as unknown as ReturnType; +const UNSAFE = prisma.$queryRawUnsafe as unknown as ReturnType; +const USER_FIND_UNIQUE = + prisma.user.findUnique as unknown as ReturnType; const MEASUREMENT_FIND_FIRST = prisma.measurement.findFirst as unknown as ReturnType; const ROLLUP_FIND_MANY = prisma.measurementRollup.findMany as unknown as ReturnType; const ROLLUP_FIND_FIRST = prisma.measurementRollup.findFirst as unknown as ReturnType; +const CANONICAL_FROM = + canonicalMeasurementsFrom as unknown as ReturnType; beforeEach(() => { RAW.mockReset(); + UNSAFE.mockReset(); + USER_FIND_UNIQUE.mockReset(); MEASUREMENT_FIND_FIRST.mockReset(); ROLLUP_FIND_MANY.mockReset(); ROLLUP_FIND_FIRST.mockReset(); + // clear (not reset) — preserve the FROM-clause stub implementation, + // drop cross-test call history so the cap assertion only sees this + // test's calls. + CANONICAL_FROM.mockClear(); + // null → loadUserSourcePriority returns null → default rank ladders. + USER_FIND_UNIQUE.mockResolvedValue(null); ROLLUP_FIND_MANY.mockResolvedValue([]); ROLLUP_FIND_FIRST.mockResolvedValue(null); MEASUREMENT_FIND_FIRST.mockResolvedValue(null); @@ -54,15 +105,14 @@ afterEach(() => { describe("computeSummariesSlice", () => { describe("cold fallback — empty rollup table", () => { it("returns the empty-summary skeleton when the user has no rows", async () => { - // v1.4.48 M0 — cold path now issues two aggregate queries - // (`allTime` + `windowed`) instead of one fat aggregate, so the - // $queryRaw call count is 4 instead of 3: - // 1. per-type coverage probe — empty ⇒ cold path. - // 2. all-time aggregate ($queryRaw) — empty. - // 3. windowed aggregate ($queryRaw, 90-day cap) — empty. - // 4. latests ($queryRaw) — empty. - RAW.mockResolvedValueOnce([]) - .mockResolvedValueOnce([]) + // v1.11.1 — the coverage probe stays on `$queryRaw` (1 RAW call); + // the three data-aggregate queries moved to `$queryRawUnsafe`: + // 1. per-type coverage probe ($queryRaw) — empty ⇒ cold path. + // 2. all-time aggregate ($queryRawUnsafe) — empty. + // 3. windowed aggregate ($queryRawUnsafe, 90-day cap) — empty. + // 4. latests ($queryRawUnsafe) — empty. + RAW.mockResolvedValueOnce([]); + UNSAFE.mockResolvedValueOnce([]) .mockResolvedValueOnce([]) .mockResolvedValueOnce([]); @@ -85,7 +135,9 @@ describe("computeSummariesSlice", () => { avg30LastYear: null, }); expect(result.bmi).toBeNull(); - expect(RAW).toHaveBeenCalledTimes(4); + // 1 RAW coverage probe + 3 UNSAFE data queries. + expect(RAW).toHaveBeenCalledTimes(1); + expect(UNSAFE).toHaveBeenCalledTimes(3); }); it("maps a populated heavy aggregate row into the DataSummary shape on cold path", async () => { @@ -95,16 +147,16 @@ describe("computeSummariesSlice", () => { // 2. all-time aggregate (count / min / max / mean) // 3. windowed aggregate (avg7/30 + slope/r²) // 4. latests - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - count: BigInt(42), - min_value: 79.2, - max_value: 84.1, - mean_value: 82.05, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: false }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + count: BigInt(42), + min_value: 79.2, + max_value: 84.1, + mean_value: 82.05, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", @@ -153,16 +205,16 @@ describe("computeSummariesSlice", () => { }); it("returns a null slope tuple when the SQL slope is null (insufficient rows)", async () => { - RAW.mockResolvedValueOnce([{ type: "PULSE", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "PULSE", - count: BigInt(1), - min_value: 72, - max_value: 72, - mean_value: 72, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "PULSE", has_buckets: false }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "PULSE", + count: BigInt(1), + min_value: 72, + max_value: 72, + mean_value: 72, + }, + ]) .mockResolvedValueOnce([ { type: "PULSE", @@ -188,16 +240,16 @@ describe("computeSummariesSlice", () => { it("surfaces lastSeenByType from the DISTINCT ON pass's measured_at", async () => { const tenDaysAgo = new Date(Date.now() - 10 * 24 * 60 * 60 * 1000); - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - count: BigInt(5), - min_value: 80, - max_value: 84, - mean_value: 82, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: false }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + count: BigInt(5), + min_value: 80, + max_value: 84, + mean_value: 82, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", @@ -225,16 +277,16 @@ describe("computeSummariesSlice", () => { }); it("seeds the latest value from the DISTINCT ON pass per type", async () => { - RAW.mockResolvedValueOnce([{ type: "PULSE", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "PULSE", - count: BigInt(3), - min_value: 60, - max_value: 95, - mean_value: 77, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "PULSE", has_buckets: false }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "PULSE", + count: BigInt(3), + min_value: 60, + max_value: 95, + mean_value: 77, + }, + ]) .mockResolvedValueOnce([ { type: "PULSE", @@ -260,29 +312,29 @@ describe("computeSummariesSlice", () => { describe("rollup-fresh happy path", () => { it("composes count/min/max/mean from the per-type rollup GROUP BY without running the heavy aggregate", async () => { - // v1.4.37.2 — the slim slice's rollup read is now a per-type - // GROUP BY ($queryRaw) instead of a row-per-bucket findMany, - // so the mock sequence is: - // 1. per-type coverage probe — WEIGHT fully covered ⇒ happy path. - // 2. narrow aggregate — windowed/regression only. - // 3. latests. - // 4. rollup GROUP BY — one row per type with count/min/max/mean - // already composed server-side. - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - avg7: 82, - avg30: 82.5, - median: 82.1, - slope7: 0.02, - r2_7: 0.5, - slope30: 0.01, - r2_30: 0.4, - slope90: 0.005, - r2_90: 0.2, - }, - ]) + // v1.11.1 — the per-type coverage probe stays on `$queryRaw`; the + // three rollup-path data queries moved to `$queryRawUnsafe`: + // 1. per-type coverage probe ($queryRaw) — WEIGHT fully covered + // ⇒ happy path. + // 2. narrow aggregate ($queryRawUnsafe) — windowed/regression only. + // 3. latests ($queryRawUnsafe). + // 4. rollup GROUP BY ($queryRawUnsafe) — one row per type with + // count/min/max/mean already composed server-side. + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + avg7: 82, + avg30: 82.5, + median: 82.1, + slope7: 0.02, + r2_7: 0.5, + slope30: 0.01, + r2_30: 0.4, + slope90: 0.005, + r2_90: 0.2, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", value: 82.7, measured_at: new Date() }, ]) @@ -320,9 +372,11 @@ describe("computeSummariesSlice", () => { confidence: 0.5, }); - // probe + narrow aggregate + latests + rollup GROUP BY - // (v1.4.37.2 — the prior `findMany` is gone). No heavy aggregate. - expect(RAW).toHaveBeenCalledTimes(4); + // 1 RAW coverage probe + 3 UNSAFE data queries (narrow aggregate + // + latests + rollup GROUP BY; v1.4.37.2 — the prior `findMany` + // is gone). No heavy aggregate. + expect(RAW).toHaveBeenCalledTimes(1); + expect(UNSAFE).toHaveBeenCalledTimes(3); // v1.4.40 W-WMY-WIRE — the year-ago baseline probe runs // `readBestGranularityRollups(userId, type, 395)` per // type-with-data via `prisma.measurementRollup.findMany`. The @@ -343,20 +397,20 @@ describe("computeSummariesSlice", () => { describe("year-over-year wiring (avg30LastYear)", () => { it("populates avg30LastYear from MONTH buckets that overlap the year-ago slice", async () => { // Coverage probe shows WEIGHT covered → rollup happy path. - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - avg7: 82, - avg30: 82.5, - slope7: 0, - r2_7: 0, - slope30: 0, - r2_30: 0, - slope90: 0, - r2_90: 0, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + avg7: 82, + avg30: 82.5, + slope7: 0, + r2_7: 0, + slope30: 0, + r2_30: 0, + slope90: 0, + r2_90: 0, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", value: 82.7, measured_at: new Date() }, ]) @@ -392,20 +446,20 @@ describe("computeSummariesSlice", () => { }); it("leaves avg30LastYear null when no bucket overlaps the year-ago slice", async () => { - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - avg7: 82, - avg30: 82.5, - slope7: 0, - r2_7: 0, - slope30: 0, - r2_30: 0, - slope90: 0, - r2_90: 0, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + avg7: 82, + avg30: 82.5, + slope7: 0, + r2_7: 0, + slope30: 0, + r2_30: 0, + slope90: 0, + r2_90: 0, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", value: 82.7, measured_at: new Date() }, ]) @@ -437,20 +491,20 @@ describe("computeSummariesSlice", () => { }); it("leaves avg30LastYear null when every granularity misses (no coverage)", async () => { - RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - avg7: 82, - avg30: 82.5, - slope7: 0, - r2_7: 0, - slope30: 0, - r2_30: 0, - slope90: 0, - r2_90: 0, - }, - ]) + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + avg7: 82, + avg30: 82.5, + slope7: 0, + r2_7: 0, + slope30: 0, + r2_30: 0, + slope90: 0, + r2_90: 0, + }, + ]) .mockResolvedValueOnce([ { type: "WEIGHT", value: 82.7, measured_at: new Date() }, ]) @@ -472,12 +526,12 @@ describe("computeSummariesSlice", () => { it("only probes types that actually have data in the current window", async () => { // v1.4.48 M0 — empty coverage map ⇒ `isFullyCovered` returns - // false ⇒ cold-fallback path. Cold path now issues 4 raw - // queries (coverage probe + all-time + windowed + latests), - // all returning empty arrays here ⇒ no types-with-data ⇒ the - // year-ago probe must NOT fan out. - RAW.mockResolvedValueOnce([]) - .mockResolvedValueOnce([]) + // false ⇒ cold-fallback path. v1.11.1 — coverage probe stays on + // `$queryRaw`; all-time + windowed + latests run via + // `$queryRawUnsafe`. All return empty arrays here ⇒ no + // types-with-data ⇒ the year-ago probe must NOT fan out. + RAW.mockResolvedValueOnce([]); + UNSAFE.mockResolvedValueOnce([]) .mockResolvedValueOnce([]) .mockResolvedValueOnce([]); @@ -499,34 +553,53 @@ describe("computeSummariesSlice", () => { */ describe("90-day outer measured_at cap (v1.4.48 M0)", () => { it("applies the 90-day cap to narrows (rollup-fresh path) and windowed (cold-fallback path)", async () => { + // v1.11.1 — the cap-bearing data queries (narrows / windowed) + // now run via `$queryRawUnsafe(sql, userId)`, so the SQL is a + // plain string arg[0] rather than a tagged-template + // strings array. Capture from the UNSAFE mock. The coverage probe + // (still `$queryRaw`) drives path selection per call. const queries: string[] = []; - RAW.mockImplementation((strings: TemplateStringsArray) => { - queries.push(strings.join("?")); + UNSAFE.mockImplementation((sql: string) => { + queries.push(sql); return Promise.resolve([]); }); // Trigger the rollup-fresh path so we capture the `narrows` SQL. // Coverage probe returns one covered type ⇒ `isFullyCovered` // is true ⇒ `computeFromRollups` runs. - RAW.mockImplementationOnce((strings: TemplateStringsArray) => { - queries.push(strings.join("?")); - return Promise.resolve([{ type: "WEIGHT", has_buckets: true }]); - }); + // Then the cold-fallback path (empty coverage) so we capture the + // `windowed` SQL. + RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]); await computeSummariesSlice("user-rollup-pin"); - // Trigger the cold-fallback path so we capture the `windowed` - // SQL. Empty coverage map ⇒ `isFullyCovered` is false ⇒ - // `computeFromLiveAggregate` runs. + RAW.mockResolvedValueOnce([]); await computeSummariesSlice("user-cold-pin"); const joined = queries.join("\n---\n"); - // Both windowed scans must carry the outer 90-day cap. - const capMatches = joined.match( - /AND m\."measured_at" >= NOW\(\) - INTERVAL '90 days'/g, + // v1.11.1 — the rollup-fresh `narrows` query still writes its + // outer 90-day cap inline, now on the canonical-source subquery's + // raw alias (`mm.`) rather than the outer `m.`. Pin that the cap + // survives the source-rank refactor. + const narrowsCap = joined.match( + /AND mm\."measured_at" >= NOW\(\) - INTERVAL '90 days'/g, + ); + expect(narrowsCap).not.toBeNull(); + expect(narrowsCap?.length).toBeGreaterThanOrEqual(1); + + // v1.11.1 — the cold-fallback `windowed` scan delegates its outer + // 90-day cap to `canonicalMeasurementsFrom(rank, "90 days")`. The + // helper lives in `@/lib/analytics/source-rank-sql` (stubbed + // above), so pin the contract at the call boundary: the slice + // must ask for the 90-day window. The `allTime` aggregate + // deliberately calls it WITHOUT an interval (no cap — all-time + // count/min/max/mean must scan every row). + expect(canonicalMeasurementsFrom).toHaveBeenCalledWith( + expect.any(String), + "90 days", + ); + expect(canonicalMeasurementsFrom).toHaveBeenCalledWith( + expect.any(String), ); - expect(capMatches).not.toBeNull(); - // narrows (rollup-fresh) + windowed (cold-fallback) = 2 caps. - expect(capMatches?.length).toBeGreaterThanOrEqual(2); }); }); }); diff --git a/src/lib/analytics/source-rank-sql.ts b/src/lib/analytics/source-rank-sql.ts new file mode 100644 index 000000000..5aac2e1c4 --- /dev/null +++ b/src/lib/analytics/source-rank-sql.ts @@ -0,0 +1,149 @@ +/** + * v1.11.1 — source-aware SQL collapse helpers. + * + * The rollup writer mints one row per (type, day, source); the live + * measurement reads see one raw row per reading. For overlapping standard + * vitals (e.g. WHOOP + Apple Watch resting heart rate) both the rollup + * all-time aggregate and the live 90-day aggregates must resolve to the + * source-priority ladder's canonical source per (type, day) before composing, + * or a dual-source user double-counts / blends two devices. + * + * `collapseRollupRowsBySource` does this in application code for the row-by-row + * readers. These helpers do the equivalent IN SQL so the slim slice + the + * comprehensive aggregator + the live `/api/measurements` fallback keep their + * single-round-trip `GROUP BY type` shape (no six-figure row transfer). + * + * Every spliced literal is a closed-enum value (`MeasurementType` / + * `MeasurementSource`) asserted against `/^[A-Z0-9_]+$/`, the same whitelist + * convention the rollup writer uses for its date-trunc + type-list splices. + * The user id is always passed as a bound parameter, never spliced. + */ +import type { MeasurementType } from "@/generated/prisma/client"; +import { metricKeyForType } from "@/lib/measurements/cumulative-day-sum"; +import { + getSourceLadder, + parseSourcePriority, +} from "@/lib/validations/source-priority"; + +/** + * Measurement types that carry a source-priority ladder — the overlapping + * vitals plus the cumulative metrics. Single-source types are intentionally + * absent: they only ever have one source per (type, day), so the collapse is + * a no-op for them and they fall into the CASE's ELSE branch. + */ +const RANKED_TYPES: readonly MeasurementType[] = [ + "RESTING_HEART_RATE", + "HEART_RATE_VARIABILITY", + "RESPIRATORY_RATE", + "OXYGEN_SATURATION", + "BODY_TEMPERATURE", + "SKIN_TEMPERATURE", + "WEIGHT", + "BODY_FAT", + "BLOOD_PRESSURE_SYS", + "BLOOD_PRESSURE_DIA", + "PULSE", + "VO2_MAX", + "RECOVERY_SCORE", + "SLEEP_DURATION", + "ACTIVITY_STEPS", + "ACTIVE_ENERGY_BURNED", + "WALKING_RUNNING_DISTANCE", + "FLIGHTS_CLIMBED", +]; + +// Enum members can carry digits (e.g. VO2_MAX, BLOOD_PRESSURE_SYS) — allow +// 0-9 so the whitelist doesn't reject a legitimate closed-enum value. +const ENUM_RE = /^[A-Z0-9_]+$/; +function assertEnumLiteral(value: string): string { + if (!ENUM_RE.test(value)) { + throw new Error(`unsafe enum literal for SQL splice: ${value}`); + } + return value; +} + +/** + * Build a SQL CASE expression mapping `(typeCol, sourceCol)` to an integer + * rank where 0 = the highest-priority source on that type's ladder. A source + * absent from a type's ladder — and every row of a type without a ladder — + * gets rank 90 so a deterministic tiebreak (`source` name, or `count DESC` + * where available) decides. Resolves the user's `sourcePriorityJson` ladders; + * `null` yields the default ladders. + */ +export function buildSourceRankCase( + priorityJson: unknown, + typeCol: string, + sourceCol: string, +): string { + const resolved = parseSourcePriority(priorityJson); + const branches: string[] = []; + for (const type of RANKED_TYPES) { + const metricKey = metricKeyForType(type); + if (!metricKey) continue; + const ladder = getSourceLadder(resolved, metricKey); + if (ladder.length === 0) continue; + const whens = ladder + .map((source, i) => `WHEN '${assertEnumLiteral(source)}' THEN ${i}`) + .join(" "); + branches.push( + `WHEN ${typeCol} = '${assertEnumLiteral(type)}' THEN ` + + `(CASE ${sourceCol} ${whens} ELSE 90 END)`, + ); + } + if (branches.length === 0) return "90"; + return `CASE ${branches.join(" ")} ELSE 90 END`; +} + +const INTERVAL_RE = /^\d+ (day|days|month|months|year|years)$/; +function assertInterval(value: string): string { + if (!INTERVAL_RE.test(value)) { + throw new Error(`unsafe interval literal for SQL splice: ${value}`); + } + return value; +} + +/** + * Build a FROM-clause subquery (aliased `m`) that restricts raw `measurements` + * to the canonical-source rows per (type, day): the inner `DISTINCT ON` picks + * the ladder-winning source for each day, and the join keeps only that source's + * readings. Use it in place of `FROM measurements m` so a live aggregate over + * an overlapping vital never blends two devices — the same collapse the rollup + * readers apply, kept in lockstep for live/rollup parity. + * + * The user id is bound as `$1`. `rankUnqualified` must be a CASE built with + * unqualified `"type"`/`"source"` columns. `sinceInterval` (e.g. `"90 days"`) + * is whitelisted and, when given, scopes both the inner pick and the outer + * filter to a trailing window. + */ +export function canonicalMeasurementsFrom( + rankUnqualified: string, + sinceInterval?: string, +): string { + const sinceInner = sinceInterval + ? `AND "measured_at" >= NOW() - INTERVAL '${assertInterval(sinceInterval)}'` + : ""; + const sinceOuter = sinceInterval + ? `AND mm."measured_at" >= NOW() - INTERVAL '${assertInterval(sinceInterval)}'` + : ""; + return `( + SELECT mm.* + FROM measurements mm + JOIN ( + SELECT DISTINCT ON ("type", date_trunc('day', "measured_at")) + "type" AS t, + date_trunc('day', "measured_at") AS d, + "source" AS canon + FROM measurements + WHERE "user_id" = $1 + AND "deleted_at" IS NULL + ${sinceInner} + ORDER BY "type", date_trunc('day', "measured_at"), (${rankUnqualified}), "source" + ) c + ON c.t = mm."type" + AND c.d = date_trunc('day', mm."measured_at") + AND c.canon = mm."source" + WHERE mm."user_id" = $1 + AND mm."deleted_at" IS NULL + ${sinceOuter} + ) m`; +} diff --git a/src/lib/analytics/summaries-slice.ts b/src/lib/analytics/summaries-slice.ts index 9a1ce8988..1c5f16dc6 100644 --- a/src/lib/analytics/summaries-slice.ts +++ b/src/lib/analytics/summaries-slice.ts @@ -62,6 +62,11 @@ import type { DataSummary } from "@/lib/analytics/trends"; import { measurementTypeEnum } from "@/lib/validations/measurement"; import { annotate } from "@/lib/logging/context"; import { ensureUserRollupsFresh } from "@/lib/rollups/measurement-rollups"; +import { loadUserSourcePriority } from "@/lib/rollups/measurement-read"; +import { + buildSourceRankCase, + canonicalMeasurementsFrom, +} from "@/lib/analytics/source-rank-sql"; import { isFullyCovered, probeRollupCoverage, @@ -274,8 +279,22 @@ export async function computeSummariesSlice( * multi-second cold to sub-second; output is bit-identical. */ async function computeFromRollups(userId: string): Promise { + // v1.11.1 — resolve the user's source-priority ladders once, then build the + // rank CASE expressions used to collapse overlapping sources to the canonical + // reading per (type, day) inside each query. `"type"`/`"source"` variants for + // the unqualified rollup + inner-subquery columns, `m."…"` for the latest + // read's outer alias. + const priorityJson = await loadUserSourcePriority(userId); + const rankUnqualified = buildSourceRankCase(priorityJson, '"type"', '"source"'); + const rankM = buildSourceRankCase(priorityJson, 'm."type"', 'm."source"'); + const [narrows, latests, dayBuckets] = await Promise.all([ - prisma.$queryRaw` + // The FROM is restricted to the canonical-source rows per (type, day): the + // inner DISTINCT ON picks the ladder-winning source for each day, and the + // join keeps only that source's readings so the 90-day AVG / median / slope + // never blend two devices that both reported the same vital. + prisma.$queryRawUnsafe( + ` SELECT m."type"::text AS type, AVG(m."value") FILTER ( @@ -329,22 +348,47 @@ async function computeFromRollups(userId: string): Promise { ) FILTER ( WHERE m."measured_at" >= NOW() - INTERVAL '90 days' )::double precision AS r2_90 - FROM measurements m - WHERE m."user_id" = ${userId} - AND m."deleted_at" IS NULL - AND m."measured_at" >= NOW() - INTERVAL '90 days' + FROM ( + SELECT mm.* + FROM measurements mm + JOIN ( + SELECT DISTINCT ON ("type", date_trunc('day', "measured_at")) + "type" AS t, + date_trunc('day', "measured_at") AS d, + "source" AS canon + FROM measurements + WHERE "user_id" = $1 + AND "deleted_at" IS NULL + AND "measured_at" >= NOW() - INTERVAL '90 days' + ORDER BY "type", date_trunc('day', "measured_at"), (${rankUnqualified}), "source" + ) c + ON c.t = mm."type" + AND c.d = date_trunc('day', mm."measured_at") + AND c.canon = mm."source" + WHERE mm."user_id" = $1 + AND mm."deleted_at" IS NULL + AND mm."measured_at" >= NOW() - INTERVAL '90 days' + ) m GROUP BY m."type" `, - prisma.$queryRaw` + userId, + ), + // v1.11.1 — the latest tile reflects the canonical source for the latest + // day, matching the chart: order by latest day first, then the ladder rank + // (canonical source wins), then the latest reading of that source. + prisma.$queryRawUnsafe( + ` SELECT DISTINCT ON (m."type") m."type"::text AS type, m."value"::double precision AS value, m."measured_at" AS measured_at FROM measurements m - WHERE m."user_id" = ${userId} + WHERE m."user_id" = $1 AND m."deleted_at" IS NULL - ORDER BY m."type", m."measured_at" DESC + ORDER BY m."type", date_trunc('day', m."measured_at") DESC, (${rankM}), m."measured_at" DESC `, + userId, + ), // v1.4.37.2 hotfix — the v1.4.35 implementation read EVERY DAY // rollup bucket for the user (`findMany` without a `bucketStart` // window) and then composed `count / min / max / mean` in JS. @@ -353,11 +397,26 @@ async function computeFromRollups(userId: string): Promise { // cache miss, even with the rollup table hot. The slim slice's // contract is the all-time count / min / max / mean per type — // exactly what a SQL `GROUP BY type` returns in a single - // round-trip. Returns 8 rows - // instead of 306k, brings the cache-miss cost into the < 100 ms - // budget. The downstream `aggregateBuckets` call is bypassed - // because the per-type aggregate is already shaped server-side. - prisma.$queryRaw` + // round-trip. + // + // v1.11.1 — rows are now per source. Collapse each (type, day) to the + // ladder-canonical source via DISTINCT ON before the all-time aggregate, + // so a dual-source vital is counted once. Still one server-side pass — + // the DISTINCT ON + GROUP BY runs in Postgres and returns one row/type. + prisma.$queryRawUnsafe( + ` + WITH collapsed AS ( + SELECT DISTINCT ON ("type", "bucket_start") + "type" AS type, + "count" AS count, + "min_value" AS min_value, + "max_value" AS max_value, + "mean" AS mean + FROM "measurement_rollups" + WHERE "user_id" = $1 + AND "granularity" = 'DAY' + ORDER BY "type", "bucket_start", (${rankUnqualified}), "source" + ) SELECT "type"::text AS type, SUM("count")::int AS count, @@ -367,11 +426,11 @@ async function computeFromRollups(userId: string): Promise { SUM("count" * "mean")::double precision / NULLIF(SUM("count")::double precision, 0) ) AS mean - FROM "measurement_rollups" - WHERE "user_id" = ${userId} - AND "granularity" = 'DAY' + FROM collapsed GROUP BY "type" `, + userId, + ), ]); const latestByType = new Map(); @@ -506,20 +565,31 @@ async function computeFromRollups(userId: string): Promise { async function computeFromLiveAggregate( userId: string, ): Promise { + // v1.11.1 — collapse overlapping sources to the ladder-canonical reading per + // (type, day) BEFORE aggregating, exactly as the rollup path does, so a + // coverage-miss tenant gets the same numbers as a warm one (live/rollup + // parity). `canonicalMeasurementsFrom` swaps `FROM measurements m` for a + // canonical-source-filtered subquery; the latest read uses the ladder rank. + const priorityJson = await loadUserSourcePriority(userId); + const rankUnqualified = buildSourceRankCase(priorityJson, '"type"', '"source"'); + const rankM = buildSourceRankCase(priorityJson, 'm."type"', 'm."source"'); + const [allTime, windowed, latests] = await Promise.all([ - prisma.$queryRaw` + prisma.$queryRawUnsafe( + ` SELECT m."type"::text AS type, COUNT(*) AS count, MIN(m."value")::double precision AS min_value, MAX(m."value")::double precision AS max_value, AVG(m."value")::double precision AS mean_value - FROM measurements m - WHERE m."user_id" = ${userId} - AND m."deleted_at" IS NULL + FROM ${canonicalMeasurementsFrom(rankUnqualified)} GROUP BY m."type" `, - prisma.$queryRaw` + userId, + ), + prisma.$queryRawUnsafe( + ` SELECT m."type"::text AS type, AVG(m."value") FILTER ( @@ -573,22 +643,24 @@ async function computeFromLiveAggregate( ) FILTER ( WHERE m."measured_at" >= NOW() - INTERVAL '90 days' )::double precision AS r2_90 - FROM measurements m - WHERE m."user_id" = ${userId} - AND m."deleted_at" IS NULL - AND m."measured_at" >= NOW() - INTERVAL '90 days' + FROM ${canonicalMeasurementsFrom(rankUnqualified, "90 days")} GROUP BY m."type" `, - prisma.$queryRaw` + userId, + ), + prisma.$queryRawUnsafe( + ` SELECT DISTINCT ON (m."type") m."type"::text AS type, m."value"::double precision AS value, m."measured_at" AS measured_at FROM measurements m - WHERE m."user_id" = ${userId} + WHERE m."user_id" = $1 AND m."deleted_at" IS NULL - ORDER BY m."type", m."measured_at" DESC + ORDER BY m."type", date_trunc('day', m."measured_at") DESC, (${rankM}), m."measured_at" DESC `, + userId, + ), ]); const latestByType = new Map(); diff --git a/src/lib/insights/__tests__/comprehensive-aggregator.test.ts b/src/lib/insights/__tests__/comprehensive-aggregator.test.ts index 86d87dc4f..3a62e54f4 100644 --- a/src/lib/insights/__tests__/comprehensive-aggregator.test.ts +++ b/src/lib/insights/__tests__/comprehensive-aggregator.test.ts @@ -14,6 +14,17 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; vi.mock("@/lib/db", () => ({ prisma: { $queryRaw: vi.fn(), + // v1.11.1 — the data-aggregate queries inside `buildFromRollups` + // and `buildFromLiveAggregate` now splice a whitelisted source-rank + // CASE and bind `userId` as `$1`, so the narrow/heavy aggregate + + // the latests pass run via `$queryRawUnsafe(sql, userId)` rather + // than the tagged-template `$queryRaw`. The coverage probe and the + // first_at query stay on `$queryRaw`. + $queryRawUnsafe: vi.fn(), + // v1.11.1 — `loadUserSourcePriority` reads the user's + // `sourcePriorityJson` to build the rank ladders. `null` here → + // default ladders. + user: { findUnique: vi.fn() }, measurement: { findMany: vi.fn(), findFirst: vi.fn() }, // v1.4.36 — the rollup read inside the Promise.all + the // freshness watermark inside `ensureUserRollupsFresh`. We mock @@ -29,10 +40,39 @@ vi.mock("@/lib/logging/context", () => ({ annotate: vi.fn(), })); +// v1.11.1 — `buildFromRollups` / `buildFromLiveAggregate` build a +// source-rank CASE via `@/lib/analytics/source-rank-sql` and splice it +// into the data-aggregate SQL. The builder's own correctness is pinned +// in its dedicated suite (and the integration suite runs the real SQL); +// here we stub it to deterministic, side-effect-free fragments so the +// aggregator's plumbing — path selection, bpRaw threading, dailyByType +// composition — is exercised without coupling the unit test to the rank +// builder's enum-whitelist internals. +vi.mock("@/lib/analytics/source-rank-sql", () => ({ + buildSourceRankCase: vi.fn(() => "90"), + canonicalMeasurementsFrom: vi.fn( + (_rank: string, sinceInterval?: string) => + `( + SELECT mm.* + FROM measurements mm + WHERE mm."user_id" = $1 + AND mm."deleted_at" IS NULL + ${ + sinceInterval + ? `AND mm."measured_at" >= NOW() - INTERVAL '${sinceInterval}'` + : "" + } + ) m`, + ), +})); + import { prisma } from "@/lib/db"; import { buildComprehensiveAggregate } from "../comprehensive-aggregator"; const RAW = prisma.$queryRaw as unknown as ReturnType; +const UNSAFE = prisma.$queryRawUnsafe as unknown as ReturnType; +const USER_FIND_UNIQUE = + prisma.user.findUnique as unknown as ReturnType; const FIND_MANY = prisma.measurement.findMany as unknown as ReturnType; const MEASUREMENT_FIND_FIRST = @@ -44,10 +84,14 @@ const ROLLUP_FIND_FIRST = beforeEach(() => { RAW.mockReset(); + UNSAFE.mockReset(); + USER_FIND_UNIQUE.mockReset(); FIND_MANY.mockReset(); MEASUREMENT_FIND_FIRST.mockReset(); ROLLUP_FIND_MANY.mockReset(); ROLLUP_FIND_FIRST.mockReset(); + // null → loadUserSourcePriority returns null → default rank ladders. + USER_FIND_UNIQUE.mockResolvedValue(null); // `ensureUserRollupsFresh` calls `prisma.measurement.findFirst` and // `prisma.measurementRollup.findFirst` in parallel. Default both to // null so the warm-up is a no-op; individual tests opt in to a @@ -65,41 +109,47 @@ describe("buildComprehensiveAggregate", () => { describe("rollup-fresh happy path", () => { it("skips the heavy live aggregate when the rollup table is populated", async () => { const now = new Date(); - // 1. per-type coverage probe — WEIGHT fully covered ⇒ happy path. - // 2. narrow aggregate ($queryRaw) — windowed/regression columns only. - // 3. latests ($queryRaw). - // 4. firstMeasurementAt ($queryRaw). + // v1.11.1 — the coverage probe + the first_at query stay on + // `$queryRaw` (2 RAW calls); the narrow aggregate + the latests + // pass moved to `$queryRawUnsafe`: + // RAW: 1. per-type coverage probe — WEIGHT covered ⇒ happy path. + // 2. firstMeasurementAt. + // UNSAFE: 1. narrow aggregate — windowed/regression columns only. + // 2. latests. RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: true }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - count: BigInt(42), - stddev_value: 1.2, - anomaly_count: BigInt(3), - avg7: 81.9, - avg30: 82.1, - avg30_last_month: 83.0, - slope7: -0.014, - r2_7: 0.65, - slope30: -0.005, - r2_30: 0.42, - slope90: 0.001, - r2_90: 0.12, - }, - ]) - .mockResolvedValueOnce([ - { type: "WEIGHT", value: 81.4, measured_at: now }, - ]) .mockResolvedValueOnce([ { first_at: new Date(now.getTime() - 86400000) }, ]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + count: BigInt(42), + stddev_value: 1.2, + anomaly_count: BigInt(3), + avg7: 81.9, + avg30: 82.1, + avg30_last_month: 83.0, + slope7: -0.014, + r2_7: 0.65, + slope30: -0.005, + r2_30: 0.42, + slope90: 0.001, + r2_90: 0.12, + }, + ]).mockResolvedValueOnce([ + { type: "WEIGHT", value: 81.4, measured_at: now }, + ]); // DAY buckets compose to count=42, min=79.2, max=84.1, mean=82.05. // The summary's count/min/max/mean read from these buckets — NOT - // from a heavy live aggregate column. + // from a heavy live aggregate column. v1.11.1 — each bucket row + // now carries a `source` so `partitionBucketsByType` → + // `collapseRollupRowsBySource` can resolve dual-source days; with + // a single source per day every bucket passes through unchanged. ROLLUP_FIND_MANY.mockResolvedValueOnce([ { type: "WEIGHT", + source: "APPLE_HEALTH", bucketStart: new Date("2026-05-10T00:00:00.000Z"), count: 20, mean: 81.0, @@ -108,6 +158,7 @@ describe("buildComprehensiveAggregate", () => { }, { type: "WEIGHT", + source: "APPLE_HEALTH", bucketStart: new Date("2026-05-11T00:00:00.000Z"), count: 22, mean: 83.0, @@ -151,12 +202,12 @@ describe("buildComprehensiveAggregate", () => { ]); // Contract pin — the heavy aggregate path is NOT exercised on the - // rollup-fresh branch. The $queryRaw calls land on the COUNT - // probe + narrow aggregate + DISTINCT-ON latest + first_at, NOT - // the legacy heavy COUNT/MIN/MAX/AVG query. We assert by call - // count + by checking that the narrow projection (no min_value / - // max_value / mean_value cols) is what landed. - expect(RAW).toHaveBeenCalledTimes(4); + // rollup-fresh branch. v1.11.1 — the coverage probe + first_at + // stay on `$queryRaw` (2 calls); the narrow aggregate + the + // DISTINCT-ON latest moved to `$queryRawUnsafe` (2 calls). Neither + // path runs the legacy heavy COUNT/MIN/MAX/AVG query. + expect(RAW).toHaveBeenCalledTimes(2); + expect(UNSAFE).toHaveBeenCalledTimes(2); // v1.4.38 W-F — sys + dia merged into a single round-trip. expect(FIND_MANY).toHaveBeenCalledTimes(1); expect(ROLLUP_FIND_MANY).toHaveBeenCalledTimes(1); @@ -165,13 +216,14 @@ describe("buildComprehensiveAggregate", () => { describe("cold fallback when no rollup buckets exist", () => { it("returns an empty bundle for a user with no measurements and no rollups", async () => { - // 1. per-type coverage probe — empty (no measurements) ⇒ cold path. - // 2. heavy aggregate ($queryRaw) — empty. - // 3. latests ($queryRaw) — empty. + // v1.11.1 — the coverage probe stays on `$queryRaw`; the heavy + // aggregate + latests moved to `$queryRawUnsafe`: + // RAW: 1. per-type coverage probe — empty ⇒ cold path. + // UNSAFE: 1. heavy aggregate — empty. + // 2. latests — empty. // No firstMeasurementAt query when totalMeasurements === 0. - RAW.mockResolvedValueOnce([]) - .mockResolvedValueOnce([]) - .mockResolvedValueOnce([]); + RAW.mockResolvedValueOnce([]); + UNSAFE.mockResolvedValueOnce([]).mockResolvedValueOnce([]); // v1.4.38 W-F — sys + dia merged into one `findMany`. FIND_MANY.mockResolvedValueOnce([]); @@ -183,7 +235,8 @@ describe("buildComprehensiveAggregate", () => { expect(result.dailyByType).toEqual({}); expect(result.firstMeasurementAt).toBeNull(); expect(result.totalMeasurements).toBe(0); - expect(RAW).toHaveBeenCalledTimes(3); + expect(RAW).toHaveBeenCalledTimes(1); + expect(UNSAFE).toHaveBeenCalledTimes(2); // v1.4.38 W-F — sys + dia merged → single findMany. expect(FIND_MANY).toHaveBeenCalledTimes(1); // The cold path's rollup.findMany still fires (in case some @@ -193,37 +246,39 @@ describe("buildComprehensiveAggregate", () => { it("runs the heavy aggregate when no rollup rows exist yet", async () => { const now = new Date(); - // 1. per-type coverage probe — WEIGHT measured but no buckets ⇒ cold path. - // 2. heavy aggregate ($queryRaw) — populated. - // 3. latests ($queryRaw) — populated. - // 4. firstMeasurementAt ($queryRaw). + // v1.11.1 — the coverage probe + first_at stay on `$queryRaw`; the + // heavy aggregate + latests moved to `$queryRawUnsafe`: + // RAW: 1. per-type coverage probe — WEIGHT measured but no + // buckets ⇒ cold path. + // 2. firstMeasurementAt (totalMeasurements > 0). + // UNSAFE: 1. heavy aggregate — populated. + // 2. latests — populated. RAW.mockResolvedValueOnce([{ type: "WEIGHT", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "WEIGHT", - count: BigInt(42), - min_value: 79.2, - max_value: 84.1, - mean_value: 82.05, - stddev_value: 1.2, - anomaly_count: BigInt(3), - avg7: 81.9, - avg30: 82.1, - avg30_last_month: 83.0, - slope7: -0.014, - r2_7: 0.65, - slope30: -0.005, - r2_30: 0.42, - slope90: 0.001, - r2_90: 0.12, - }, - ]) - .mockResolvedValueOnce([ - { type: "WEIGHT", value: 81.4, measured_at: now }, - ]) .mockResolvedValueOnce([ { first_at: new Date(now.getTime() - 86400000) }, ]); + UNSAFE.mockResolvedValueOnce([ + { + type: "WEIGHT", + count: BigInt(42), + min_value: 79.2, + max_value: 84.1, + mean_value: 82.05, + stddev_value: 1.2, + anomaly_count: BigInt(3), + avg7: 81.9, + avg30: 82.1, + avg30_last_month: 83.0, + slope7: -0.014, + r2_7: 0.65, + slope30: -0.005, + r2_30: 0.42, + slope90: 0.001, + r2_90: 0.12, + }, + ]).mockResolvedValueOnce([ + { type: "WEIGHT", value: 81.4, measured_at: now }, + ]); // v1.4.38 W-F — sys + dia merged into one round-trip. FIND_MANY.mockResolvedValueOnce([]); @@ -269,35 +324,37 @@ describe("buildComprehensiveAggregate", () => { it("threads sys/dia raw rows through bpRawRows so 5-min pairing survives", async () => { const measuredAt = new Date("2026-05-10T08:00:00Z"); // Cold path so the heavy aggregate fires (BP type lacks coverage). - RAW.mockResolvedValueOnce([{ type: "BLOOD_PRESSURE_SYS", has_buckets: false }]) - .mockResolvedValueOnce([ - { - type: "BLOOD_PRESSURE_SYS", - count: BigInt(1), - min_value: 120, - max_value: 120, - mean_value: 120, - stddev_value: 0, - anomaly_count: BigInt(0), - avg7: 120, - avg30: 120, - avg30_last_month: null, - slope7: null, - r2_7: null, - slope30: null, - r2_30: null, - slope90: null, - r2_90: null, - }, - ]) - .mockResolvedValueOnce([ - { - type: "BLOOD_PRESSURE_SYS", - value: 120, - measured_at: measuredAt, - }, - ]) - .mockResolvedValueOnce([{ first_at: measuredAt }]); + // v1.11.1 — coverage probe + first_at on `$queryRaw`; heavy + latests + // on `$queryRawUnsafe`. + RAW.mockResolvedValueOnce([ + { type: "BLOOD_PRESSURE_SYS", has_buckets: false }, + ]).mockResolvedValueOnce([{ first_at: measuredAt }]); + UNSAFE.mockResolvedValueOnce([ + { + type: "BLOOD_PRESSURE_SYS", + count: BigInt(1), + min_value: 120, + max_value: 120, + mean_value: 120, + stddev_value: 0, + anomaly_count: BigInt(0), + avg7: 120, + avg30: 120, + avg30_last_month: null, + slope7: null, + r2_7: null, + slope30: null, + r2_30: null, + slope90: null, + r2_90: null, + }, + ]).mockResolvedValueOnce([ + { + type: "BLOOD_PRESSURE_SYS", + value: 120, + measured_at: measuredAt, + }, + ]); // v1.4.38 W-F — single merged `findMany` returning both sys + dia // rows tagged by `type`. The aggregator partitions in JS, so the // bpRawRows.sys / .dia shape stays byte-identical. diff --git a/src/lib/insights/__tests__/weight-status.test.ts b/src/lib/insights/__tests__/weight-status.test.ts index 878364eb0..07b454a62 100644 --- a/src/lib/insights/__tests__/weight-status.test.ts +++ b/src/lib/insights/__tests__/weight-status.test.ts @@ -6,6 +6,10 @@ vi.mock("@/lib/db", () => ({ measurement: { findMany: vi.fn() }, measurementRollup: { findMany: vi.fn() }, moodEntry: { findMany: vi.fn() }, + // v1.11.1 — the rollup readers lazy-load the user's + // `sourcePriorityJson` via `loadUserSourcePriority`. `null` here → + // default rank ladders. + user: { findUnique: vi.fn() }, }, })); @@ -47,6 +51,8 @@ beforeEach(() => { // Cold rollup tier: the graded builder folds monthly/yearly from the // full-history `measurement.findMany` fallback the test already mocks. vi.mocked(prisma.measurementRollup.findMany).mockResolvedValue([] as never); + // v1.11.1 — null source-priority blob → default rank ladders. + vi.mocked(prisma.user.findUnique).mockResolvedValue(null as never); }); describe("generateWeightStatusForUser — graded payload", () => { diff --git a/src/lib/insights/comprehensive-aggregator.ts b/src/lib/insights/comprehensive-aggregator.ts index 10ec3e72d..bbf6aa5bb 100644 --- a/src/lib/insights/comprehensive-aggregator.ts +++ b/src/lib/insights/comprehensive-aggregator.ts @@ -100,8 +100,20 @@ import { prisma } from "@/lib/db"; import type { DataSummary } from "@/lib/analytics/trends"; import { annotate } from "@/lib/logging/context"; +import type { + MeasurementSource, + MeasurementType, +} from "@/generated/prisma/client"; import { ensureUserRollupsFresh } from "@/lib/rollups/measurement-rollups"; -import { aggregateBuckets } from "@/lib/rollups/measurement-read"; +import { + aggregateBuckets, + collapseRollupRowsBySource, + loadUserSourcePriority, +} from "@/lib/rollups/measurement-read"; +import { + buildSourceRankCase, + canonicalMeasurementsFrom, +} from "@/lib/analytics/source-rank-sql"; import { isFullyCovered, probeRollupCoverage, @@ -339,17 +351,24 @@ async function buildFromRollups( ): Promise { // v1.4.38 W-F — bp sys + dia consolidated into one `findMany` // (`type: { in: [...] }`) plus per-sub-query wall-clock timings. + // v1.11.1 — collapse overlapping sources to the ladder-canonical reading per + // (type, day) before aggregating. `rankUnqualified` ranks the inner + // canonical-source subquery; `rankM` ranks the latest read's `m.`-aliased + // columns. The window_stats CTE + the main narrow both read from the + // canonical-source-filtered set so a dual-source vital isn't blended. + const priorityJson = await loadUserSourcePriority(userId); + const rankUnqualified = buildSourceRankCase(priorityJson, '"type"', '"source"'); + const rankM = buildSourceRankCase(priorityJson, 'm."type"', 'm."source"'); + const [narrows, latests, dayBuckets, bpRawRows, firstRows] = await Promise.all([ - timeSubquery(timings, "narrow", () => prisma.$queryRaw` + timeSubquery(timings, "narrow", () => prisma.$queryRawUnsafe( + ` WITH window_stats AS ( SELECT m."type", AVG(m."value") AS mean_value, STDDEV_POP(m."value") AS stddev_value - FROM measurements m - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} - AND m."deleted_at" IS NULL + FROM ${canonicalMeasurementsFrom(rankUnqualified, "90 days")} GROUP BY m."type" ) SELECT @@ -412,24 +431,26 @@ async function buildFromRollups( ) FILTER ( WHERE m."measured_at" >= NOW() - INTERVAL '90 days' )::double precision AS r2_90 - FROM measurements m + FROM ${canonicalMeasurementsFrom(rankUnqualified, "90 days")} JOIN window_stats ws ON ws."type" = m."type" - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} - AND m."deleted_at" IS NULL GROUP BY m."type", ws.stddev_value - `), - timeSubquery(timings, "latests", () => prisma.$queryRaw` + `, + userId, + )), + timeSubquery(timings, "latests", () => prisma.$queryRawUnsafe( + ` SELECT DISTINCT ON (m."type") m."type"::text AS type, m."value"::double precision AS value, m."measured_at" AS measured_at FROM measurements m - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} + WHERE m."user_id" = $1 + AND m."measured_at" >= NOW() - INTERVAL '90 days' AND m."deleted_at" IS NULL - ORDER BY m."type", m."measured_at" DESC - `), + ORDER BY m."type", date_trunc('day', m."measured_at") DESC, (${rankM}), m."measured_at" DESC + `, + userId, + )), // v1.4.35 — DAY rollup buckets for the trailing 90 days, every // type. Drives both the per-type `count / min / max / mean` // composition (via `aggregateBuckets`) and the `dailyByType` @@ -466,7 +487,7 @@ async function buildFromRollups( }); } - const bucketsByType = partitionBucketsByType(dayBuckets); + const bucketsByType = partitionBucketsByType(dayBuckets, priorityJson); const narrowByType = new Map(); for (const row of narrows) { narrowByType.set(row.type, row); @@ -536,17 +557,21 @@ async function buildFromLiveAggregate( // v1.4.38 W-F — same sub-query timing + consolidated bp-raw read as // the rollup-fresh path so the prod observability covers cold mounts // too. + // v1.11.1 — same source-aware collapse as the rollup-fresh path so the cold + // start agrees with the warm path (live/rollup parity). + const priorityJson = await loadUserSourcePriority(userId); + const rankUnqualified = buildSourceRankCase(priorityJson, '"type"', '"source"'); + const rankM = buildSourceRankCase(priorityJson, 'm."type"', 'm."source"'); + const [aggregates, latests, dayBuckets, bpRawRows] = await Promise.all([ - timeSubquery(timings, "heavy", () => prisma.$queryRaw` + timeSubquery(timings, "heavy", () => prisma.$queryRawUnsafe( + ` WITH window_stats AS ( SELECT m."type", AVG(m."value") AS mean_value, STDDEV_POP(m."value") AS stddev_value - FROM measurements m - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} - AND m."deleted_at" IS NULL + FROM ${canonicalMeasurementsFrom(rankUnqualified, "90 days")} GROUP BY m."type" ) SELECT @@ -609,24 +634,26 @@ async function buildFromLiveAggregate( ) FILTER ( WHERE m."measured_at" >= NOW() - INTERVAL '90 days' )::double precision AS r2_90 - FROM measurements m + FROM ${canonicalMeasurementsFrom(rankUnqualified, "90 days")} JOIN window_stats ws ON ws."type" = m."type" - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} - AND m."deleted_at" IS NULL GROUP BY m."type", ws.stddev_value - `), - timeSubquery(timings, "latests", () => prisma.$queryRaw` + `, + userId, + )), + timeSubquery(timings, "latests", () => prisma.$queryRawUnsafe( + ` SELECT DISTINCT ON (m."type") m."type"::text AS type, m."value"::double precision AS value, m."measured_at" AS measured_at FROM measurements m - WHERE m."user_id" = ${userId} - AND m."measured_at" >= ${ninetyDaysAgo} + WHERE m."user_id" = $1 + AND m."measured_at" >= NOW() - INTERVAL '90 days' AND m."deleted_at" IS NULL - ORDER BY m."type", m."measured_at" DESC - `), + ORDER BY m."type", date_trunc('day', m."measured_at") DESC, (${rankM}), m."measured_at" DESC + `, + userId, + )), // Buckets may exist for some types even when the COUNT probe came // back zero (race window between the probe and a sibling // populator); the cold-path read still honours them for the @@ -651,7 +678,7 @@ async function buildFromLiveAggregate( }); } - const bucketsByType = partitionBucketsByType(dayBuckets); + const bucketsByType = partitionBucketsByType(dayBuckets, priorityJson); const summaries: Record = {}; let totalMeasurements = 0; @@ -713,12 +740,14 @@ async function buildFromLiveAggregate( function partitionBucketsByType( rows: ReadonlyArray<{ type: string; + source: MeasurementSource; bucketStart: Date; count: number; mean: number; minValue: number; maxValue: number; }>, + userPriorityJson: unknown, ): Map< string, Array<{ @@ -729,6 +758,16 @@ function partitionBucketsByType( maxValue: number; }> > { + // v1.11.1 — rollup rows are per source. Group by type, then collapse each + // type's buckets to the ladder-canonical source per day before composing, + // so a dual-source vital is not double-counted in count/mean/dailyByType. + const byType = new Map>(); + for (const b of rows) { + const list = byType.get(b.type) ?? []; + list.push(b); + byType.set(b.type, list); + } + const map = new Map< string, Array<{ @@ -739,16 +778,22 @@ function partitionBucketsByType( maxValue: number; }> >(); - for (const b of rows) { - const list = map.get(b.type) ?? []; - list.push({ - day: b.bucketStart, - count: b.count, - mean: b.mean, - minValue: b.minValue, - maxValue: b.maxValue, - }); - map.set(b.type, list); + for (const [type, typeRows] of byType) { + const collapsed = collapseRollupRowsBySource( + typeRows, + type as MeasurementType, + userPriorityJson, + ); + map.set( + type, + collapsed.map((b) => ({ + day: b.bucketStart, + count: b.count, + mean: b.mean, + minValue: b.minValue, + maxValue: b.maxValue, + })), + ); } return map; } diff --git a/src/lib/jobs/__tests__/coach-memory-refresh-queue.test.ts b/src/lib/jobs/__tests__/coach-memory-refresh-queue.test.ts new file mode 100644 index 000000000..c52b8eada --- /dev/null +++ b/src/lib/jobs/__tests__/coach-memory-refresh-queue.test.ts @@ -0,0 +1,46 @@ +/** + * v1.11.1 — queue-registration guard for the combined Coach memory-refresh + * job (rolling conversation summary + durable fact extraction). + * + * Same source-text-grep approach as the period-narrative / stress-strain + * guards: assert the queue is imported, registered in `allQueues`, and wired + * to a `boss.work` handler that runs the refresh pipeline — without booting + * pg-boss + Prisma. An unregistered queue silently never runs (the recurring + * past bug this guards against; v1.4.37 W10 caught exactly this class). + */ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { describe, expect, it } from "vitest"; + +const REMINDER_WORKER_PATH = join(__dirname, "..", "reminder-worker.ts"); +const source = readFileSync(REMINDER_WORKER_PATH, "utf8"); + +function allQueuesBlock(): string { + const m = source.match(/const allQueues\s*=\s*\[([\s\S]*?)\];/); + expect(m).not.toBeNull(); + return m![1]; +} + +describe("reminder-worker — coach-memory-refresh wiring", () => { + it("imports the queue symbol + worker from the coach-memory modules", () => { + expect(source).toMatch( + /from\s*["']@\/lib\/ai\/coach\/coach-memory-shared["']/, + ); + expect(source).toMatch(/\bCOACH_MEMORY_REFRESH_QUEUE\b/); + expect(source).toMatch( + /from\s*["']@\/lib\/ai\/coach\/coach-memory-refresh-worker["']/, + ); + expect(source).toMatch(/\brunCoachMemoryRefresh\b/); + }); + + it("registers the coach-memory-refresh queue in allQueues", () => { + expect(allQueuesBlock()).toMatch(/\bCOACH_MEMORY_REFRESH_QUEUE\b/); + }); + + it("registers a boss.work handler that runs the refresh pipeline", () => { + expect(source).toMatch( + /boss\.work[\s\S]{0,200}COACH_MEMORY_REFRESH_QUEUE[\s\S]{0,400}runCoachMemoryRefresh/, + ); + }); +}); diff --git a/src/lib/jobs/reminder-worker.ts b/src/lib/jobs/reminder-worker.ts index 7fdf857fb..5ac070485 100644 --- a/src/lib/jobs/reminder-worker.ts +++ b/src/lib/jobs/reminder-worker.ts @@ -102,6 +102,11 @@ import { warmOneNarrative, type PeriodNarrativePayload, } from "@/lib/jobs/period-narrative-warm"; +import { + COACH_MEMORY_REFRESH_QUEUE, + type CoachMemoryRefreshPayload, +} from "@/lib/ai/coach/coach-memory-shared"; +import { runCoachMemoryRefresh } from "@/lib/ai/coach/coach-memory-refresh-worker"; import { DENSE_INTRADAY_RETENTION_QUEUE, DENSE_INTRADAY_RETENTION_CONCURRENCY, @@ -2329,6 +2334,11 @@ export async function startReminderWorker() { // the read-only narrative GET. The queue MUST be registered here or the // GET-miss enqueue silently never warms. PERIOD_NARRATIVE_QUEUE, + // v1.11.1 — combined Coach memory-refresh (rolling conversation summary + + // durable fact extraction), enqueued fire-and-forget from a long chat + // turn. The queue MUST be registered here or the enqueue silently never + // runs. + COACH_MEMORY_REFRESH_QUEUE, ]; for (const q of allQueues) { @@ -2737,6 +2747,26 @@ export async function startReminderWorker() { } }, ); + // v1.11.1 — combined Coach memory refresh: rolling conversation summary + + // durable fact extraction for one long conversation. localConcurrency 1 so a + // burst of long-conversation turns can't fan out concurrent provider calls; + // each step is budget-gated inside runStatusCompletion and fault-isolated. + await boss.work( + COACH_MEMORY_REFRESH_QUEUE, + { localConcurrency: 1 }, + async (jobs) => { + for (const job of jobs) { + if (!job.data?.conversationId || !job.data?.userId) continue; + try { + await runCoachMemoryRefresh(job.data); + } catch (err) { + recordError(); + workerLog("error", "[coach-memory-refresh] failed", err); + throw err; + } + } + }, + ); // v1.8.3 — on-demand per-metric status generation enqueued by the // read-only status route on a cold card. Low concurrency so a first // visit that cold-misses several cards can't saturate the Prisma pool diff --git a/src/lib/measurements/cumulative-day-sum.ts b/src/lib/measurements/cumulative-day-sum.ts index 3c09c056c..c67eea962 100644 --- a/src/lib/measurements/cumulative-day-sum.ts +++ b/src/lib/measurements/cumulative-day-sum.ts @@ -131,3 +131,50 @@ export function cumulativeMetricKey( return null; } } + +/** + * v1.11.1 — full MeasurementType → SourcePriorityMetricKey map for the + * source-aware rollup collapse. A superset of `cumulativeMetricKey`: it adds + * the overlapping NON-cumulative vitals (spot / daily readings that two or + * more sources realistically report for the same day) so the rollup read path + * can resolve the canonical source through the same ladder the raw-row picker + * uses. Cumulative types fall through to `cumulativeMetricKey`. Returns null + * for single-source types (no competing source today → source-blind grouping + * is already correct) and for types without a ladder — the collapse treats a + * null key as "no priority, keep one row deterministically". + */ +export function metricKeyForType( + type: MeasurementType, +): SourcePriorityMetricKey | null { + switch (type) { + case "RESTING_HEART_RATE": + return "restingHeartRate"; + case "HEART_RATE_VARIABILITY": + return "hrv"; + case "RESPIRATORY_RATE": + return "respiratoryRate"; + case "OXYGEN_SATURATION": + return "spo2"; + case "BODY_TEMPERATURE": + return "bodyTemperature"; + case "SKIN_TEMPERATURE": + return "skinTemperature"; + case "WEIGHT": + return "weight"; + case "BODY_FAT": + return "bodyFat"; + case "BLOOD_PRESSURE_SYS": + case "BLOOD_PRESSURE_DIA": + return "bloodPressure"; + case "PULSE": + return "pulse"; + case "VO2_MAX": + return "vo2Max"; + case "RECOVERY_SCORE": + return "recovery"; + case "SLEEP_DURATION": + return "sleep"; + default: + return cumulativeMetricKey(type); + } +} diff --git a/src/lib/openapi/routes.ts b/src/lib/openapi/routes.ts index 5d35f7df2..5e146fedb 100644 --- a/src/lib/openapi/routes.ts +++ b/src/lib/openapi/routes.ts @@ -45,6 +45,7 @@ import { } from "@/lib/validations/medication"; import { medicationExtractionSchema } from "@/lib/ai/coach/medication-extract-prompt"; import { ACCEPTED_INSIGHTS_TILE_IDS } from "@/lib/insights-layout"; +import { COACH_FACT_CATEGORIES } from "@/lib/ai/coach/facts"; import { exportSelectionSchema } from "@/lib/validations/health-record-export"; import { createShareLinkSchema } from "@/lib/validations/clinician-share-link"; import { METRIC_STATUS_IDS } from "@/lib/insights/metric-status-registry"; @@ -2010,6 +2011,49 @@ const stdResponses = { }, }; +// ── Coach facts (v1.11.1) ──────────────────────────────────────────── +// Read + delete surface for the durable facts the Coach extracts. Facts +// are server-extracted, not user-authored, so there is no create/update +// shape — only list, bulk-clear, and single-delete responses. + +const coachFactItem = z.object({ + id: z.string(), + category: z + .enum(COACH_FACT_CATEGORIES) + .describe( + "App-side closed category: preference | condition | goal | constraint | context.", + ), + text: z.string().describe("Decrypted fact text."), + confidence: z + .number() + .int() + .describe("0..100 server-assigned extraction confidence."), + createdAt: z.iso.datetime({ offset: true }), +}); + +const coachFactsListResponse = z.object({ + facts: z + .array(coachFactItem) + .describe( + "The caller's active facts, highest-confidence then newest first. Undecryptable rows are omitted.", + ), +}); + +const coachFactsClearedResponse = z.object({ + cleared: z + .number() + .int() + .describe("Number of active facts soft-deleted by the bulk clear."), +}); + +const coachFactDeletedResponse = z.object({ + deleted: z + .boolean() + .describe( + "True when a fact owned by the caller was soft-deleted; false for an unknown / cross-user / already-deleted id (idempotent no-op).", + ), +}); + // ── Path table ─────────────────────────────────────────────────────── export const openApiPaths: NonNullable = { @@ -3228,6 +3272,68 @@ export const openApiPaths: NonNullable = { }, }, }, + "/api/insights/coach/facts": { + get: { + tags: ["Insights"], + summary: "List the caller's durable Coach facts", + description: + "v1.11.1 — returns the active facts the Coach has extracted about the caller (highest-confidence then newest first), each decrypted on the fly. The GDPR 'what do you know about me' surface. Coach-gated (`requireAssistantSurface(\"coach\")`). Auth via cookie or Bearer; the owner is always narrowed from the session, never the body. Undecryptable rows are omitted rather than failing the read.", + responses: { + "200": { + description: "The caller's active facts.", + content: { + "application/json": { + schema: dataEnvelope(coachFactsListResponse, "CoachFactsList"), + }, + }, + }, + ...stdResponses, + }, + }, + delete: { + tags: ["Insights"], + summary: "Forget all of the caller's Coach facts", + description: + "v1.11.1 — bulk 'forget what you know about me': soft-deletes every active fact for the caller and returns the count cleared. Idempotent (a second call clears 0). Coach-gated. Auth via cookie or Bearer.", + responses: { + "200": { + description: "All active facts cleared; the count is returned.", + content: { + "application/json": { + schema: dataEnvelope( + coachFactsClearedResponse, + "CoachFactsCleared", + ), + }, + }, + }, + ...stdResponses, + }, + }, + }, + "/api/insights/coach/facts/{id}": { + delete: { + tags: ["Insights"], + summary: "Forget one Coach fact", + description: + "v1.11.1 — soft-deletes a single fact owned by the caller. An unknown / cross-user / already-deleted id is an idempotent no-op returning `{ deleted: false }`, never revealing whether the id exists under another account. Coach-gated. Auth via cookie or Bearer.", + responses: { + "200": { + description: + "The fact was soft-deleted (`deleted: true`) or the id matched nothing the caller owns (`deleted: false`).", + content: { + "application/json": { + schema: dataEnvelope( + coachFactDeletedResponse, + "CoachFactDeleted", + ), + }, + }, + }, + ...stdResponses, + }, + }, + }, "/api/insights/chat/messages/{id}/feedback": { post: { tags: ["Insights"], diff --git a/src/lib/rollups/__tests__/collapse-rollup-rows-by-source.test.ts b/src/lib/rollups/__tests__/collapse-rollup-rows-by-source.test.ts new file mode 100644 index 000000000..3108cae1b --- /dev/null +++ b/src/lib/rollups/__tests__/collapse-rollup-rows-by-source.test.ts @@ -0,0 +1,159 @@ +/** + * v1.11.1 — source-aware rollup collapse. + * + * The writer mints one rollup row per (type, day, source). This helper + * resolves overlapping sources to ONE row per bucket via the user's + * source-priority ladder, so a day with WHOOP + Apple Watch resting heart + * rate surfaces the ladder-canonical reading instead of an AVG blend. + * + * Default ladders pinned here (from `DEFAULT_SOURCE_PRIORITY`): + * restingHeartRate: WHOOP > APPLE_HEALTH > WITHINGS + * spo2: WITHINGS > WHOOP > APPLE_HEALTH > MANUAL + * recovery: WHOOP > COMPUTED + * steps: APPLE_HEALTH > WITHINGS > MANUAL + */ +import { describe, expect, it } from "vitest"; + +import type { MeasurementSource } from "@/generated/prisma/client"; +import { collapseRollupRowsBySource } from "../measurement-read"; + +interface Row { + bucketStart: Date; + source: MeasurementSource; + count: number; + mean: number; + sumValue: number | null; +} + +const DAY1 = new Date("2026-06-01T00:00:00.000Z"); +const DAY2 = new Date("2026-06-02T00:00:00.000Z"); + +function row( + bucketStart: Date, + source: MeasurementSource, + mean: number, + count = 1, + sumValue: number | null = null, +): Row { + return { bucketStart, source, count, mean, sumValue }; +} + +describe("collapseRollupRowsBySource", () => { + it("picks WHOOP over Apple for resting heart rate (ladder head wins)", () => { + const out = collapseRollupRowsBySource( + [ + row(DAY1, "APPLE_HEALTH", 54), + row(DAY1, "WHOOP", 51), + ], + "RESTING_HEART_RATE", + null, + ); + expect(out).toHaveLength(1); + expect(out[0].source).toBe("WHOOP"); + expect(out[0].mean).toBe(51); + }); + + it("picks Withings over Apple for SpO₂", () => { + const out = collapseRollupRowsBySource( + [ + row(DAY1, "APPLE_HEALTH", 96), + row(DAY1, "WITHINGS", 98), + ], + "OXYGEN_SATURATION", + null, + ); + expect(out).toHaveLength(1); + expect(out[0].source).toBe("WITHINGS"); + expect(out[0].mean).toBe(98); + }); + + it("picks native WHOOP over the COMPUTED proxy for RECOVERY_SCORE", () => { + const out = collapseRollupRowsBySource( + [ + row(DAY1, "COMPUTED", 62), + row(DAY1, "WHOOP", 70), + ], + "RECOVERY_SCORE", + null, + ); + expect(out).toHaveLength(1); + expect(out[0].source).toBe("WHOOP"); + }); + + it("collapses cumulative steps to ONE source's sum (never cross-source sum)", () => { + const out = collapseRollupRowsBySource( + [ + row(DAY1, "WITHINGS", 3800, 1, 3800), + row(DAY1, "APPLE_HEALTH", 4000, 1, 4000), + ], + "ACTIVITY_STEPS", + null, + ); + expect(out).toHaveLength(1); + // steps ladder = APPLE_HEALTH first → canonical source's sum only, NOT 7800. + expect(out[0].source).toBe("APPLE_HEALTH"); + expect(out[0].sumValue).toBe(4000); + }); + + it("preserves multi-day series, collapsing each day independently", () => { + const out = collapseRollupRowsBySource( + [ + row(DAY1, "APPLE_HEALTH", 54), + row(DAY1, "WHOOP", 51), + row(DAY2, "WHOOP", 49), + ], + "RESTING_HEART_RATE", + null, + ); + expect(out).toHaveLength(2); + expect(out.map((r) => r.bucketStart.getTime())).toEqual([ + DAY1.getTime(), + DAY2.getTime(), + ]); + expect(out.every((r) => r.source === "WHOOP")).toBe(true); + }); + + it("leaves a single-source day unchanged (fast path)", () => { + const input = [row(DAY1, "WITHINGS", 72)]; + const out = collapseRollupRowsBySource(input, "WEIGHT", null); + expect(out).toBe(input); + }); + + it("honours a custom source-priority override", () => { + // User pins Apple above WHOOP for resting heart rate. + const custom = { + restingHeartRate: ["APPLE_HEALTH", "WHOOP", "WITHINGS"], + }; + const out = collapseRollupRowsBySource( + [ + row(DAY1, "WHOOP", 51), + row(DAY1, "APPLE_HEALTH", 54), + ], + "RESTING_HEART_RATE", + custom, + ); + expect(out).toHaveLength(1); + expect(out[0].source).toBe("APPLE_HEALTH"); + }); + + it("falls back to the alphabetically-smallest source when none is on the ladder", () => { + // Neither IMPORT nor MANUAL is on the restingHeartRate ladder; keep one + // row deterministically by source name, matching the live-SQL paths' + // `ORDER BY … source` tiebreak (live/rollup parity). + const out = collapseRollupRowsBySource( + [ + row(DAY1, "MANUAL", 58, 5), + row(DAY1, "IMPORT", 60, 2), + ], + "RESTING_HEART_RATE", + null, + ); + expect(out).toHaveLength(1); + // "IMPORT" < "MANUAL" alphabetically. + expect(out[0].source).toBe("IMPORT"); + }); + + it("returns empty input untouched", () => { + expect(collapseRollupRowsBySource([], "WEIGHT", null)).toEqual([]); + }); +}); diff --git a/src/lib/rollups/__tests__/measurement-read-wmy.test.ts b/src/lib/rollups/__tests__/measurement-read-wmy.test.ts index 04531c6fb..f47c6dffa 100644 --- a/src/lib/rollups/__tests__/measurement-read-wmy.test.ts +++ b/src/lib/rollups/__tests__/measurement-read-wmy.test.ts @@ -24,6 +24,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; const mocks = vi.hoisted(() => ({ findMany: vi.fn(), + userFindUnique: vi.fn(), })); vi.mock("@/lib/db", () => ({ @@ -31,6 +32,12 @@ vi.mock("@/lib/db", () => ({ measurementRollup: { findMany: mocks.findMany, }, + // v1.11.1 — readBestGranularityRollups loads the source-priority blob via + // loadUserSourcePriority; default to null so the collapse uses the default + // ladders (these routing fixtures are single-source per bucket anyway). + user: { + findUnique: mocks.userFindUnique, + }, }, })); @@ -62,6 +69,8 @@ function bucket( beforeEach(() => { findMany.mockReset(); + mocks.userFindUnique.mockReset(); + mocks.userFindUnique.mockResolvedValue({ sourcePriorityJson: null }); }); afterEach(() => { diff --git a/src/lib/rollups/__tests__/measurement-rollups.test.ts b/src/lib/rollups/__tests__/measurement-rollups.test.ts index 7a5891898..f96c0018e 100644 --- a/src/lib/rollups/__tests__/measurement-rollups.test.ts +++ b/src/lib/rollups/__tests__/measurement-rollups.test.ts @@ -23,6 +23,7 @@ const mocks = vi.hoisted(() => ({ queryRawUnsafe: vi.fn(), transaction: vi.fn(), upsert: vi.fn(), + createMany: vi.fn(), deleteMany: vi.fn(), findFirst: vi.fn(), findMany: vi.fn(), @@ -38,6 +39,7 @@ vi.mock("@/lib/db", () => ({ $transaction: mocks.transaction, measurementRollup: { upsert: mocks.upsert, + createMany: mocks.createMany, deleteMany: mocks.deleteMany, findFirst: mocks.findFirst, findMany: mocks.findMany, @@ -66,6 +68,7 @@ const { queryRawUnsafe, transaction, upsert, + createMany, deleteMany, findFirst, findFirstMeasurement, @@ -96,6 +99,7 @@ beforeEach(() => { queryRawUnsafe.mockReset(); transaction.mockReset(); upsert.mockReset(); + createMany.mockReset(); deleteMany.mockReset(); findFirst.mockReset(); mocks.findMany.mockReset(); @@ -107,9 +111,14 @@ beforeEach(() => { // assertion. _resetEnsureUserRollupsFreshInFlightForTests(); // Default: $transaction takes an array of pre-built promises (the - // populator passes `slice.map(prisma.measurementRollup.upsert(...))`) - // and returns them awaited. - transaction.mockImplementation(async (operations: unknown[]) => operations); + // populator passes `[deleteMany(...), createMany(...)]`) and resolves + // them like the real client so the destructured create result is the + // resolved value, not a pending promise. + transaction.mockImplementation(async (operations: unknown[]) => + Promise.all(operations), + ); + // v1.11.1 — persistRollupRows writes via createMany; default the count. + createMany.mockResolvedValue({ count: 1 }); }); afterEach(() => { @@ -146,10 +155,11 @@ describe("collapseToTypeDayKeys", () => { }); describe("recomputeBucketsForMeasurement", () => { - it("upserts the DAY rollup synchronously and enqueues WEEK/MONTH/YEAR", async () => { + it("writes the DAY rollup synchronously and enqueues WEEK/MONTH/YEAR", async () => { queryRawUnsafe.mockResolvedValueOnce([ { type: "WEIGHT", + source: "MANUAL", bucket_start: new Date("2026-05-10T00:00:00.000Z"), count: BigInt(3), mean: 82.5, @@ -173,26 +183,24 @@ describe("recomputeBucketsForMeasurement", () => { new Date("2026-05-10T14:30:00.000Z"), ); - // DAY pass — single upsert in a transaction. + // DAY pass — v1.11.1 delete-then-insert the day partition in one tx. expect(transaction).toHaveBeenCalledTimes(1); - expect(upsert).toHaveBeenCalledTimes(1); - const upsertArg = upsert.mock.calls[0][0]; - expect(upsertArg.where.userId_type_granularity_bucketStart.userId).toBe( - "user-1", - ); - expect(upsertArg.where.userId_type_granularity_bucketStart.type).toBe( - "WEIGHT", - ); - expect( - upsertArg.where.userId_type_granularity_bucketStart.granularity, - ).toBe("DAY"); - expect(upsertArg.create.count).toBe(3); - expect(upsertArg.create.mean).toBe(82.5); - // v1.4.39 W-SUM — sumValue flows through on both create + update - // halves of the upsert. Cumulative read paths (steps, flights, - // distance, daylight, active-energy) consume this directly. - expect(upsertArg.create.sumValue).toBe(247.5); - expect(upsertArg.update.sumValue).toBe(247.5); + expect(deleteMany).toHaveBeenCalledTimes(1); + expect(deleteMany.mock.calls[0][0].where.userId).toBe("user-1"); + expect(deleteMany.mock.calls[0][0].where.granularity).toBe("DAY"); + expect(deleteMany.mock.calls[0][0].where.type.in).toContain("WEIGHT"); + expect(createMany).toHaveBeenCalledTimes(1); + const created = createMany.mock.calls[0][0].data; + expect(created).toHaveLength(1); + expect(created[0].userId).toBe("user-1"); + expect(created[0].type).toBe("WEIGHT"); + expect(created[0].granularity).toBe("DAY"); + expect(created[0].source).toBe("MANUAL"); + expect(created[0].count).toBe(3); + expect(created[0].mean).toBe(82.5); + // v1.4.39 W-SUM — sumValue flows through. Cumulative read paths + // (steps, flights, distance, daylight, active-energy) consume it. + expect(created[0].sumValue).toBe(247.5); // WEEK / MONTH / YEAR — three enqueues against the worker queue. expect(bossSend).toHaveBeenCalledTimes(3); @@ -210,6 +218,7 @@ describe("recomputeBucketsForMeasurement", () => { queryRawUnsafe.mockResolvedValueOnce([ { type: "ACTIVITY_STEPS", + source: "APPLE_HEALTH", bucket_start: new Date("2026-05-10T00:00:00.000Z"), count: BigInt(5), mean: 2496, @@ -230,13 +239,13 @@ describe("recomputeBucketsForMeasurement", () => { new Date("2026-05-10T14:30:00.000Z"), ); - expect(upsert).toHaveBeenCalledTimes(1); - const arg = upsert.mock.calls[0][0]; - expect(arg.create.sumValue).toBe(12480); + expect(createMany).toHaveBeenCalledTimes(1); + const arg = createMany.mock.calls[0][0].data[0]; + expect(arg.sumValue).toBe(12480); // mean × count algebraic equivalence — the writer carries SUM // directly because the aggregator computed it once. Asserting // both gives parity coverage for the consumer-side fallback. - expect(arg.create.mean * arg.create.count).toBe(12480); + expect(arg.mean * arg.count).toBe(12480); }); it("passes through null sum_value when the aggregator returns NULL", async () => { @@ -247,6 +256,7 @@ describe("recomputeBucketsForMeasurement", () => { queryRawUnsafe.mockResolvedValueOnce([ { type: "WEIGHT", + source: "MANUAL", bucket_start: new Date("2026-05-10T00:00:00.000Z"), count: BigInt(1), mean: 82.5, @@ -266,9 +276,8 @@ describe("recomputeBucketsForMeasurement", () => { new Date("2026-05-10T14:30:00.000Z"), ); - const arg = upsert.mock.calls[0][0]; - expect(arg.create.sumValue).toBeNull(); - expect(arg.update.sumValue).toBeNull(); + const arg = createMany.mock.calls[0][0].data[0]; + expect(arg.sumValue).toBeNull(); }); it("deletes the DAY row when the post-mutation aggregate is empty", async () => { diff --git a/src/lib/rollups/measurement-read-wmy.ts b/src/lib/rollups/measurement-read-wmy.ts index dc6a3fc72..7d9222ec7 100644 --- a/src/lib/rollups/measurement-read-wmy.ts +++ b/src/lib/rollups/measurement-read-wmy.ts @@ -61,6 +61,10 @@ import type { } from "@/generated/prisma/client"; import { prisma } from "@/lib/db"; +import { + collapseRollupRowsBySource, + loadUserSourcePriority, +} from "@/lib/rollups/measurement-read"; /** * Normalised bucket row returned by every WMY reader. Mirrors the @@ -134,19 +138,32 @@ export async function readBestGranularityRollups( userId: string, type: MeasurementType, windowDays: number, + userPriorityJson?: unknown, ): Promise<{ granularity: RollupGranularity; rows: RollupBucketRow[]; } | null> { if (!Number.isFinite(windowDays) || windowDays <= 0) return null; const since = new Date(Date.now() - windowDays * 24 * 60 * 60 * 1000); + // v1.11.1 — load the source-priority blob once and thread it into every + // granularity probe so the collapse never re-queries the user per floor. + const priority = + userPriorityJson !== undefined + ? userPriorityJson + : await loadUserSourcePriority(userId); // Walk the floors from coarsest to finest and return the first // granularity whose floor the window clears AND which has coverage // for `(userId, type, since)`. The `if rows == null` fall-through // is what makes the helper resilient to partial coverage. for (const floor of GRANULARITY_FLOORS) { if (windowDays < floor.minWindowDays) continue; - const rows = await readGranularity(userId, type, floor.granularity, since); + const rows = await readGranularity( + userId, + type, + floor.granularity, + since, + priority, + ); if (rows && rows.length > 0) { return { granularity: floor.granularity, rows }; } @@ -211,8 +228,9 @@ async function readGranularity( type: MeasurementType, granularity: RollupGranularity, since: Date, + userPriorityJson: unknown, ): Promise { - // Bounded `findMany`: `(userId, type, granularity, bucketStart)` + // Bounded `findMany`: `(userId, type, granularity, bucketStart, source)` // is the composite primary key so the planner picks the index path // every time. `bucketStart >= since` is the same shape // `readRollupBuckets` uses; we don't carry an upper bound because @@ -227,6 +245,8 @@ async function readGranularity( orderBy: { bucketStart: "asc" }, select: { bucketStart: true, + // v1.11.1 — source drives the per-bucket canonical collapse below. + source: true, count: true, mean: true, sd: true, @@ -238,7 +258,9 @@ async function readGranularity( }, }); if (rows.length === 0) return null; - return rows.map((r) => ({ + // v1.11.1 — collapse overlapping sources to the ladder-canonical reading + // per bucket, then drop the source column from the normalised shape. + return collapseRollupRowsBySource(rows, type, userPriorityJson).map((r) => ({ bucketStart: r.bucketStart, count: r.count, mean: r.mean, diff --git a/src/lib/rollups/measurement-read.ts b/src/lib/rollups/measurement-read.ts index 557c20c74..67d657470 100644 --- a/src/lib/rollups/measurement-read.ts +++ b/src/lib/rollups/measurement-read.ts @@ -20,6 +20,31 @@ * the v1.4.34.1 / 4.5 aggregator survives. */ +import type { + MeasurementSource, + MeasurementType, +} from "@/generated/prisma/client"; +import { prisma } from "@/lib/db"; +import { metricKeyForType } from "@/lib/measurements/cumulative-day-sum"; +import { + getSourceLadder, + parseSourcePriority, +} from "@/lib/validations/source-priority"; + +/** + * v1.11.1 — load a user's source-priority blob for the rollup collapse. + * `null` makes `collapseRollupRowsBySource` fall back to the default ladders. + * Callers that read many types in a loop should load it once and thread it + * through, rather than paying one lookup per read. + */ +export async function loadUserSourcePriority(userId: string): Promise { + const user = await prisma.user.findUnique({ + where: { id: userId }, + select: { sourcePriorityJson: true }, + }); + return user?.sourcePriorityJson ?? null; +} + export interface DailyMeanRow { day: Date; count: number; @@ -28,6 +53,81 @@ export interface DailyMeanRow { maxValue: number; } +/** Minimal shape the source collapse needs from a per-source rollup row. */ +export interface SourcedBucketRow { + bucketStart: Date; + source: MeasurementSource; + count: number; +} + +/** + * v1.11.1 — collapse per-source rollup rows to ONE row per bucket using the + * user's source-priority ladder. The writer mints one row per + * (type, day, source); this resolves overlapping sources (e.g. WHOOP + Apple + * Watch resting heart rate) to the ladder-canonical reading before the linear + * composition in `aggregateBuckets` / the WMY readers runs. Cumulative types + * collapse to the single canonical source too, so the caller reads that one + * source's summed `sumValue` — a day's total reflects one source, never a + * cross-source sum. + * + * Resolution per bucket: + * 1. the first source in the metric's ladder that is present → canonical; + * 2. no ladder match (an unlisted source, or a type with no ladder) → the + * row with the most readings (max `count`), tie-broken by source name, + * so the bucket neither doubles nor goes dark. + * + * Input order is preserved (buckets emit in first-seen order). A single-source + * day (the common case) short-circuits to the row unchanged. + */ +export function collapseRollupRowsBySource( + rows: T[], + type: MeasurementType, + userPriorityJson: unknown, +): T[] { + if (rows.length <= 1) return rows; + + const byBucket = new Map(); + for (const row of rows) { + const key = row.bucketStart.getTime(); + const slot = byBucket.get(key); + if (slot) slot.push(row); + else byBucket.set(key, [row]); + } + + const metricKey = metricKeyForType(type); + const ladder: readonly MeasurementSource[] = metricKey + ? getSourceLadder(parseSourcePriority(userPriorityJson), metricKey) + : []; + + const out: T[] = []; + for (const bucketRows of byBucket.values()) { + if (bucketRows.length === 1) { + out.push(bucketRows[0]); + continue; + } + let picked: T | undefined; + for (const source of ladder) { + const hit = bucketRows.find((r) => r.source === source); + if (hit) { + picked = hit; + break; + } + } + if (!picked) { + // No ladder match — keep one row deterministically by alphabetically + // smallest source name, so the bucket neither doubles nor goes dark AND + // the pick matches the live-SQL paths' `ORDER BY … source` tiebreak + // (live/rollup parity for a ranked type whose day carries only + // non-ladder sources). + picked = bucketRows.reduce((best, r) => + r.source < best.source ? r : best, + ); + } + out.push(picked); + } + return out; +} + /** * Combine DAY buckets into the linearly-composable window stats — * `count`, `min`, `max`, `mean`. SD / slope / R² are intentionally diff --git a/src/lib/rollups/measurement-rollups.ts b/src/lib/rollups/measurement-rollups.ts index e8c95bc94..be973b5aa 100644 --- a/src/lib/rollups/measurement-rollups.ts +++ b/src/lib/rollups/measurement-rollups.ts @@ -30,8 +30,13 @@ import { prisma } from "@/lib/db"; import { getGlobalBoss } from "@/lib/jobs/boss-instance"; import { annotate } from "@/lib/logging/context"; +import { + collapseRollupRowsBySource, + loadUserSourcePriority, +} from "@/lib/rollups/measurement-read"; import { startOfUtcDay } from "@/lib/tz/start-of-utc-day"; import type { + MeasurementSource, MeasurementType, RollupGranularity, } from "@/generated/prisma/client"; @@ -102,6 +107,7 @@ const DATE_TRUNC_UNIT: Record = { /** Raw row shape returned by the aggregate `$queryRaw`. */ interface RollupRow { type: string; + source: string; bucket_start: Date; count: bigint; mean: number | null; @@ -118,10 +124,10 @@ interface RollupRow { * `types` / `granularities` / `from` / `to` filters; defaults cover * every type, every granularity, and the trailing 5 years. * - * Idempotent: rows are upserted under the - * `(userId, type, granularity, bucketStart)` composite primary key, - * so re-running the call is a no-op for unchanged buckets and a - * refresh for changed ones. + * Idempotent: rows are delete-then-inserted per source under the + * `(userId, type, granularity, bucketStart, source)` composite primary + * key, so re-running the call refreshes the affected buckets and drops + * any per-source row whose source no longer reports that bucket. */ export async function recomputeUserRollups( userId: string, @@ -260,6 +266,11 @@ export async function enqueueRollupRecompute(input: { /** * Read rollup rows for `(userId, type, granularity)` in `[from, to)`. * Returns rows sorted ascending by `bucketStart`. + * + * v1.11.1 — rows are stored per source; this collapses overlapping sources to + * the ladder-canonical reading so the return shape stays one row per bucket. + * Pass `userPriorityJson` to avoid a per-call user lookup when the caller + * already loaded it (e.g. a loop over many types); omit it to lazy-load. */ export async function readRollupBuckets( userId: string, @@ -267,6 +278,7 @@ export async function readRollupBuckets( granularity: RollupGranularity, from: Date, to: Date, + userPriorityJson?: unknown, ): Promise< Array<{ bucketStart: Date; @@ -289,7 +301,11 @@ export async function readRollupBuckets( }, orderBy: { bucketStart: "asc" }, }); - return rows.map((r) => ({ + const priority = + userPriorityJson !== undefined + ? userPriorityJson + : await loadUserSourcePriority(userId); + return collapseRollupRowsBySource(rows, type, priority).map((r) => ({ bucketStart: r.bucketStart, count: r.count, mean: r.mean, @@ -337,7 +353,9 @@ async function runRollupAggregate(input: { // from `MeasurementType` (Prisma-generated TS enum); we whitelist // them with a strict regex before splicing into SQL. for (const t of input.types) { - if (!/^[A-Z_]+$/.test(t)) { + // Enum members can carry digits (e.g. VO2_MAX) — allow 0-9 so a + // legitimate type is not rejected on the write-hook's typed recompute. + if (!/^[A-Z0-9_]+$/.test(t)) { throw new Error(`invalid measurement type: ${t}`); } } @@ -347,6 +365,7 @@ async function runRollupAggregate(input: { const sql = ` SELECT m."type"::text AS type, + m."source"::text AS source, ${dateTrunc} AS bucket_start, COUNT(*) AS count, AVG(m."value")::double precision AS mean, @@ -368,7 +387,7 @@ async function runRollupAggregate(input: { AND m."measured_at" >= $2 AND m."measured_at" < $3 AND m."deleted_at" IS NULL - GROUP BY m."type", ${dateTrunc} + GROUP BY m."type", m."source", ${dateTrunc} `; return prisma.$queryRawUnsafe( sql, @@ -381,6 +400,7 @@ async function runRollupAggregate(input: { const sql = ` SELECT m."type"::text AS type, + m."source"::text AS source, ${dateTrunc} AS bucket_start, COUNT(*) AS count, AVG(m."value")::double precision AS mean, @@ -401,7 +421,7 @@ async function runRollupAggregate(input: { AND m."measured_at" >= $2 AND m."measured_at" < $3 AND m."deleted_at" IS NULL - GROUP BY m."type", ${dateTrunc} + GROUP BY m."type", m."source", ${dateTrunc} `; return prisma.$queryRawUnsafe( sql, @@ -423,61 +443,77 @@ async function persistRollupRows( rows: RollupRow[], ): Promise { if (rows.length === 0) return 0; - // Chunk to keep the parameter count below Postgres' 65k cap and - // the transaction round-trip count bounded for large backfills. - // For the typical hot path (a single DAY recompute = 1 row, a - // multi-week WEEK recompute = a few rows) the round-trip count is - // negligible. + + // v1.11.1 — rows are now minted per source. Delete-then-insert the affected + // (type, bucket) partitions across ALL sources before writing, so a source + // that disappeared from a bucket (the user disconnected a provider, or + // deleted the last reading from one device) does not strand a stale + // per-source row that a plain upsert would never revisit. The delete is + // scoped to the [min, max] bucket range of the rows being written, per type + // — an indexed range delete, cheap on the single-day write hook and bounded + // on the full backfill. + const types = [...new Set(rows.map((r) => r.type as MeasurementType))]; + let minBucket = rows[0].bucket_start; + let maxBucket = rows[0].bucket_start; + for (const r of rows) { + if (r.bucket_start < minBucket) minBucket = r.bucket_start; + if (r.bucket_start > maxBucket) maxBucket = r.bucket_start; + } + const deleteWhere = { + userId, + granularity, + type: { in: types }, + bucketStart: { gte: minBucket, lte: maxBucket }, + }; + + const toData = (row: RollupRow) => ({ + userId, + type: row.type as MeasurementType, + granularity, + bucketStart: row.bucket_start, + source: row.source as MeasurementSource, + count: Number(row.count), + mean: row.mean ?? 0, + minValue: row.min_value ?? 0, + maxValue: row.max_value ?? 0, + // v1.4.39 W-SUM — populate for every type. Cumulative read paths (steps, + // flights, distance, daylight, active-energy) consume this column + // directly; spot metrics carry it for free because the underlying SUM + // aggregator runs in the same query as AVG / MIN / MAX. + sumValue: row.sum_value ?? null, + sd: row.sd, + slope: row.slope, + r2: row.r2, + computedAt: new Date(), + }); + + // Chunk to keep the parameter count below Postgres' 65k cap and the + // round-trip count bounded for large backfills. const CHUNK = 500; + + // Common path — the entire write fits one chunk (every incremental hook + // and most bucket-span recomputes). Atomic delete-then-insert so a + // concurrent read never sees a half-rewritten partition. + if (rows.length <= CHUNK) { + const [, created] = await prisma.$transaction([ + prisma.measurementRollup.deleteMany({ where: deleteWhere }), + prisma.measurementRollup.createMany({ data: rows.map(toData) }), + ]); + return created.count; + } + + // Large backfill — delete the partition range once, then insert in chunks. + // The brief gap between delete and the first insert can only make a + // concurrent reader fall back to live SQL for those buckets (correct + // value), never serve a wrong one. + await prisma.measurementRollup.deleteMany({ where: deleteWhere }); let touched = 0; for (let i = 0; i < rows.length; i += CHUNK) { const slice = rows.slice(i, i + CHUNK); - await prisma.$transaction( - slice.map((row) => - prisma.measurementRollup.upsert({ - where: { - userId_type_granularity_bucketStart: { - userId, - type: row.type as MeasurementType, - granularity, - bucketStart: row.bucket_start, - }, - }, - create: { - userId, - type: row.type as MeasurementType, - granularity, - bucketStart: row.bucket_start, - count: Number(row.count), - mean: row.mean ?? 0, - minValue: row.min_value ?? 0, - maxValue: row.max_value ?? 0, - // v1.4.39 W-SUM — populate for every type. Cumulative read - // paths (steps, flights, distance, daylight, active-energy) - // consume this column directly; spot metrics carry it for - // free because the underlying SUM aggregator runs in the - // same query as AVG / MIN / MAX. - sumValue: row.sum_value ?? null, - sd: row.sd, - slope: row.slope, - r2: row.r2, - computedAt: new Date(), - }, - update: { - count: Number(row.count), - mean: row.mean ?? 0, - minValue: row.min_value ?? 0, - maxValue: row.max_value ?? 0, - sumValue: row.sum_value ?? null, - sd: row.sd, - slope: row.slope, - r2: row.r2, - computedAt: new Date(), - }, - }), - ), - ); - touched += slice.length; + const res = await prisma.measurementRollup.createMany({ + data: slice.map(toData), + }); + touched += res.count; } return touched; } diff --git a/tests/integration/measurement-soft-delete.test.ts b/tests/integration/measurement-soft-delete.test.ts index e36810713..c33a78bbe 100644 --- a/tests/integration/measurement-soft-delete.test.ts +++ b/tests/integration/measurement-soft-delete.test.ts @@ -289,14 +289,15 @@ describe("Measurement.deletedAt — tombstone invisibility (v1.4.40 W-DELETED)", noonUtc.getUTCDate(), ), ); - const bucket = await prisma.measurementRollup.findUnique({ + // v1.11.1 — rollups are source-aware; both seeded rows share the default + // MANUAL source so they still collapse to one DAY row. findFirst keeps the + // assertion source-agnostic. + const bucket = await prisma.measurementRollup.findFirst({ where: { - userId_type_granularity_bucketStart: { - userId: user.id, - type: "WEIGHT", - granularity: "DAY", - bucketStart: dayStartUtc, - }, + userId: user.id, + type: "WEIGHT", + granularity: "DAY", + bucketStart: dayStartUtc, }, }); expect(bucket).not.toBeNull(); diff --git a/tests/integration/source-aware-rollups.integration.test.ts b/tests/integration/source-aware-rollups.integration.test.ts new file mode 100644 index 000000000..f180dab39 --- /dev/null +++ b/tests/integration/source-aware-rollups.integration.test.ts @@ -0,0 +1,407 @@ +/** + * v1.11.1 — source-aware measurement rollups, DB round-trip. + * + * The rollup grain gained `source` in its primary key + * (`@@id([userId, type, granularity, bucketStart, source])`): the populator + * now mints ONE row per (type, day, source) instead of one source-blind + * aggregate per (type, day). The read path + * (`readRollupBuckets` → `collapseRollupRowsBySource`) collapses the + * overlapping per-source rows back to ONE row per bucket using the user's + * source-priority ladder before the linear DAY-bucket composition runs. + * + * `collapseRollupRowsBySource` is unit-tested in isolation + * (`src/lib/rollups/__tests__/collapse-rollup-rows-by-source.test.ts`); this + * suite pins the same contract end-to-end through the real Postgres + * testcontainer — the populator SQL groups by `m."source"` and the reader + * lazy-loads the user's ladder from `sourcePriorityJson`. Five contracts: + * + * 1. **Point-metric ladder pick.** RHR with WHOOP + APPLE_HEALTH on one + * day → two raw rows, one collapsed row that surfaces the WHOOP value + * (WHOOP leads the restingHeartRate ladder). + * 2. **Cumulative no-blend.** ACTIVITY_STEPS with APPLE_HEALTH + WITHINGS + * on one day → two raw rows, one collapsed row reflecting ONLY the + * Apple canonical source's count/mean — never the cross-source sum. + * 3. **Native-vs-derived.** RECOVERY_SCORE with WHOOP + COMPUTED → the + * WHOOP (device-native) row wins over the COMPUTED proxy. + * 4. **Disappearing source.** Soft-deleting the WHOOP RHR rows and + * recomputing drops the stale WHOOP rollup row (delete-then-insert), + * leaving the APPLE row alone — the reader now surfaces the Apple value. + * 5. **Idempotency.** Re-running the recompute on a dual-source day keeps + * the per-source row count at two (no duplicate-per-source rows). + */ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { getPrismaClient, truncateAllTables } from "./setup"; +import { + recomputeUserRollups, + readRollupBuckets, +} from "@/lib/rollups/measurement-rollups"; + +vi.mock("@/lib/db-compat", () => ({ + ensureDbCompatibility: vi.fn().mockResolvedValue(undefined), +})); + +// `recomputeBucketsForMeasurement` enqueues WEEK/MONTH/YEAR jobs via pg-boss; +// the DAY-only `recomputeUserRollups` calls here never touch the queue, but the +// sibling rollups test mocks the boss instance and the integration suite runs +// with `isolate: false`, so we leave it detached (null) for parity. +vi.mock("@/lib/jobs/boss-instance", () => ({ + getGlobalBoss: vi.fn(() => null), +})); + +// Deterministic UTC day used across every case. All seeded `measuredAt` +// values land inside this calendar day so each (type, source) folds to a +// single DAY bucket. +const DAY = "2026-05-20"; +const DAY_START_UTC = new Date(`${DAY}T00:00:00.000Z`); +const DAY_END_UTC = new Date(`${DAY}T23:59:59.999Z`); +// Read window [from, to) that brackets the seeded day with room to spare. +const FROM = new Date(`${DAY}T00:00:00.000Z`); +const TO = new Date("2026-05-21T00:00:00.000Z"); + +let seq = 0; +async function seedUser(prisma: ReturnType) { + seq += 1; + return prisma.user.create({ + data: { + username: `source-aware-rollup-${seq}`, + email: `source-aware-rollup-${seq}@example.test`, + role: "USER", + }, + }); +} + +beforeEach(async () => { + await truncateAllTables(getPrismaClient()); +}); + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe("source-aware measurement rollups — integration (v1.11.1)", () => { + it("RESTING_HEART_RATE: keeps one raw row per source, collapses to the WHOOP ladder pick", async () => { + const prisma = getPrismaClient(); + const user = await seedUser(prisma); + + await prisma.measurement.createMany({ + data: [ + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 51, + unit: "bpm", + source: "WHOOP", + measuredAt: new Date(`${DAY}T06:00:00.000Z`), + }, + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 54, + unit: "bpm", + source: "APPLE_HEALTH", + measuredAt: new Date(`${DAY}T07:00:00.000Z`), + }, + ], + }); + + await recomputeUserRollups(user.id, { + types: ["RESTING_HEART_RATE"], + granularities: ["DAY"], + }); + + // Raw table: one row PER source for the (type, day). + const rawRows = await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: "RESTING_HEART_RATE", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + orderBy: { source: "asc" }, + }); + expect(rawRows).toHaveLength(2); + expect(rawRows.map((r) => r.source).sort()).toEqual([ + "APPLE_HEALTH", + "WHOOP", + ]); + // Each per-source row carries that source's single reading. + const apple = rawRows.find((r) => r.source === "APPLE_HEALTH")!; + const whoop = rawRows.find((r) => r.source === "WHOOP")!; + expect(apple.count).toBe(1); + expect(apple.mean).toBeCloseTo(54, 5); + expect(whoop.count).toBe(1); + expect(whoop.mean).toBeCloseTo(51, 5); + + // Read path collapses to ONE bucket — WHOOP wins the restingHeartRate + // ladder (WHOOP > APPLE_HEALTH > WITHINGS). + const buckets = await readRollupBuckets( + user.id, + "RESTING_HEART_RATE", + "DAY", + FROM, + TO, + ); + expect(buckets).toHaveLength(1); + expect(buckets[0].mean).toBeCloseTo(51, 5); + expect(buckets[0].count).toBe(1); + }); + + it("ACTIVITY_STEPS: collapses to the Apple canonical source only — never a cross-source 7800 blend", async () => { + const prisma = getPrismaClient(); + const user = await seedUser(prisma); + + await prisma.measurement.createMany({ + data: [ + { + userId: user.id, + type: "ACTIVITY_STEPS", + value: 4000, + unit: "count", + source: "APPLE_HEALTH", + measuredAt: new Date(`${DAY}T08:00:00.000Z`), + }, + { + userId: user.id, + type: "ACTIVITY_STEPS", + value: 3800, + unit: "count", + source: "WITHINGS", + measuredAt: new Date(`${DAY}T09:00:00.000Z`), + }, + ], + }); + + await recomputeUserRollups(user.id, { + types: ["ACTIVITY_STEPS"], + granularities: ["DAY"], + }); + + const rawRows = await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: "ACTIVITY_STEPS", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + orderBy: { source: "asc" }, + }); + expect(rawRows).toHaveLength(2); + const apple = rawRows.find((r) => r.source === "APPLE_HEALTH")!; + const withings = rawRows.find((r) => r.source === "WITHINGS")!; + expect(apple.mean).toBeCloseTo(4000, 5); + expect(apple.sumValue).toBeCloseTo(4000, 5); + expect(withings.mean).toBeCloseTo(3800, 5); + + // Steps ladder = APPLE_HEALTH > WITHINGS > MANUAL. The collapsed bucket + // reflects ONLY the Apple row — a single canonical source's daily total, + // not the 7800 cross-source sum that a source-blind aggregate produced. + const buckets = await readRollupBuckets( + user.id, + "ACTIVITY_STEPS", + "DAY", + FROM, + TO, + ); + expect(buckets).toHaveLength(1); + expect(buckets[0].count).toBe(1); // one Apple reading, not two summed + expect(buckets[0].mean).toBeCloseTo(4000, 5); + expect(buckets[0].mean).not.toBeCloseTo(7800, 5); + expect(buckets[0].mean).not.toBeCloseTo(3900, 5); // not a cross-source avg + }); + + it("RECOVERY_SCORE: native WHOOP wins over the COMPUTED proxy", async () => { + const prisma = getPrismaClient(); + const user = await seedUser(prisma); + + await prisma.measurement.createMany({ + data: [ + { + userId: user.id, + type: "RECOVERY_SCORE", + value: 70, + unit: "score", + source: "WHOOP", + measuredAt: new Date(`${DAY}T05:00:00.000Z`), + }, + { + userId: user.id, + type: "RECOVERY_SCORE", + value: 62, + unit: "score", + source: "COMPUTED", + measuredAt: new Date(`${DAY}T05:30:00.000Z`), + }, + ], + }); + + await recomputeUserRollups(user.id, { + types: ["RECOVERY_SCORE"], + granularities: ["DAY"], + }); + + const rawRows = await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: "RECOVERY_SCORE", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + }); + expect(rawRows).toHaveLength(2); + + // recovery ladder = WHOOP > COMPUTED — the device-native score wins. + const buckets = await readRollupBuckets( + user.id, + "RECOVERY_SCORE", + "DAY", + FROM, + TO, + ); + expect(buckets).toHaveLength(1); + expect(buckets[0].mean).toBeCloseTo(70, 5); + }); + + it("disappearing source: soft-deleting WHOOP rows drops the stale WHOOP rollup row on recompute", async () => { + const prisma = getPrismaClient(); + const user = await seedUser(prisma); + + await prisma.measurement.createMany({ + data: [ + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 51, + unit: "bpm", + source: "WHOOP", + measuredAt: new Date(`${DAY}T06:00:00.000Z`), + }, + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 54, + unit: "bpm", + source: "APPLE_HEALTH", + measuredAt: new Date(`${DAY}T07:00:00.000Z`), + }, + ], + }); + + await recomputeUserRollups(user.id, { + types: ["RESTING_HEART_RATE"], + granularities: ["DAY"], + }); + + // Both sources present after the first fold. + const firstFold = await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: "RESTING_HEART_RATE", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + }); + expect(firstFold).toHaveLength(2); + + // The WHOOP source disappears (user disconnected the strap / deleted the + // last reading from that device). Tombstone the WHOOP rows. + await prisma.measurement.updateMany({ + where: { userId: user.id, source: "WHOOP" }, + data: { deletedAt: new Date() }, + }); + + // Re-fold the same day. The populator delete-then-inserts the affected + // (type, bucket) partition across ALL sources, so the now-empty WHOOP + // bucket is dropped rather than stranded as a stale row a plain upsert + // would never revisit. + await recomputeUserRollups(user.id, { + types: ["RESTING_HEART_RATE"], + granularities: ["DAY"], + }); + + const secondFold = await prisma.measurementRollup.findMany({ + where: { + userId: user.id, + type: "RESTING_HEART_RATE", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + }); + expect(secondFold).toHaveLength(1); + expect(secondFold[0].source).toBe("APPLE_HEALTH"); + + // The reader now surfaces the Apple value — WHOOP is gone from the ladder + // resolution because it no longer has a row. + const buckets = await readRollupBuckets( + user.id, + "RESTING_HEART_RATE", + "DAY", + FROM, + TO, + ); + expect(buckets).toHaveLength(1); + expect(buckets[0].mean).toBeCloseTo(54, 5); + }); + + it("idempotency: re-running the recompute keeps the per-source row count at two", async () => { + const prisma = getPrismaClient(); + const user = await seedUser(prisma); + + await prisma.measurement.createMany({ + data: [ + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 51, + unit: "bpm", + source: "WHOOP", + measuredAt: new Date(`${DAY}T06:00:00.000Z`), + }, + { + userId: user.id, + type: "RESTING_HEART_RATE", + value: 54, + unit: "bpm", + source: "APPLE_HEALTH", + measuredAt: new Date(`${DAY}T07:00:00.000Z`), + }, + ], + }); + + const countRows = () => + prisma.measurementRollup.count({ + where: { + userId: user.id, + type: "RESTING_HEART_RATE", + granularity: "DAY", + bucketStart: { gte: DAY_START_UTC, lte: DAY_END_UTC }, + }, + }); + + await recomputeUserRollups(user.id, { + types: ["RESTING_HEART_RATE"], + granularities: ["DAY"], + }); + expect(await countRows()).toBe(2); + + // Second pass over the identical fixture must not duplicate per-source + // rows — the delete-then-insert keys on (userId, type, granularity, + // bucketStart, source). + await recomputeUserRollups(user.id, { + types: ["RESTING_HEART_RATE"], + granularities: ["DAY"], + }); + expect(await countRows()).toBe(2); + + // And the collapsed read is still a single WHOOP-canonical bucket. + const buckets = await readRollupBuckets( + user.id, + "RESTING_HEART_RATE", + "DAY", + FROM, + TO, + ); + expect(buckets).toHaveLength(1); + expect(buckets[0].mean).toBeCloseTo(51, 5); + }); +});