Add privacy-safe AgentPond tracing for AI Insights

.agents/skills/agentpond-instrumentation/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: agentpond-instrumentation
+description: Add OpenInference tracing to a Firebase server application and export spans directly to AgentPond through Firebase Storage. Use when instrumenting an untraced Firebase AI application, adding a missing OpenInference integration, or adapting an existing OpenTelemetry setup to use createFirebaseSpanExporter().
+---
+
+# AgentPond Instrumentation
+
+Instrument Firebase AI applications without changing business behavior. Analyze the target service first, reuse existing tracing infrastructure, and keep Firebase Admin and trace export strictly server-side.
+
+Read these references when relevant:
+
+- Firebase Admin, exporter, and Storage Rules: [references/firebase.md](references/firebase.md)
+- OpenInference routing, custom spans, sessions, and verification: [references/openinference.md](references/openinference.md)
+
+## Core principles
+
+- Inspect before editing. Confirm the service, runtime, package manager, AI SDK, framework, and existing telemetry.
+- Instrument only trusted server code. Never import `firebase-admin` or `@agentpond/firebase` into browser or client bundles.
+- Prefer framework or provider auto-instrumentation. Add manual spans only for application logic, chains, tools, or gaps.
+- Reuse the existing Firebase default app and global OpenTelemetry provider. Do not register a competing provider.
+- Initialize tracing before importing or constructing instrumented AI clients.
+- Keep tracing additive and follow the repository's conventions.
+- Never add credentials to source code or ask the user to paste secrets into chat.
+
+## Phase 0: preflight
+
+1. Confirm which Firebase service should be instrumented. In a monorepo, do not assume every Functions source or server package is in scope.
+2. Identify the build, typecheck, start, emulator, and real-request commands needed for verification.
+3. Confirm the target is a trusted Node.js server runtime. If only client code exists, stop and ask whether the user wants to add a server function or another trusted runtime.
+4. Read [references/firebase.md](references/firebase.md) before proposing Firebase changes.
+
+## Phase 1: read-only analysis
+
+Do not write files or install packages during this phase.
+
+1. Inspect `firebase.json`, `.firebaserc`, package manifests, lockfiles, and server entrypoints.
+2. Scan imports to identify:
+   - AI providers and clients
+   - agent or LLM frameworks
+   - existing OpenInference or OpenTelemetry setup
+   - Firebase Admin initialization
+   - request, conversation, and tool execution boundaries
+3. Find the Storage Rules file configured by `firebase.json` and review whether any matching client rule can access `agentpond/**`.
+4. Prefer a framework-native OpenInference integration when it captures model and tool spans. Add a provider instrumentor only for a documented gap.
+5. Return a concise proposal containing:
+   - target service and package manager
+   - detected AI SDKs and framework
+   - packages to install
+   - existing Firebase and telemetry initialization to reuse
+   - Storage Rules status
+   - files and verification commands expected to change
+
+Stop after presenting the proposal and ask for explicit confirmation before installing packages or editing files. The initial request to instrument the project does not replace confirmation of the analyzed target, package choices, files, and Storage Rules changes.
+
+## Phase 2: implementation
+
+1. Read current official integration documentation for the detected framework or AI client.
+2. Install packages with the project's package manager:
+   - `@agentpond/firebase`
+   - required OpenTelemetry SDK packages
+   - the matching `@arizeai/openinference-*` package
+   - `firebase-admin` only when the trusted server package does not already provide it
+3. Create or update one centralized server instrumentation module.
+4. Reuse an existing default Firebase Admin app. Add default initialization only when it is absent; follow [references/firebase.md](references/firebase.md).
+5. Create `createFirebaseSpanExporter()` after the default app exists.
+6. Add the exporter to the existing provider. When no provider exists, create one using APIs supported by the installed OpenTelemetry version and prefer NodeSDK's batched `traceExporter` configuration or an explicit `BatchSpanProcessor`.
+7. Register the selected OpenInference instrumentation before AI clients are created.
+8. Add manual CHAIN and TOOL spans only where auto-instrumentation leaves important application behavior invisible.
+9. Preserve one `session.id` across all turns in the same conversation.
+10. Fix unsafe Storage Rules before declaring instrumentation complete. Do not add a standalone false rule as a supposed override for a broader allow.
+
+## Verification
+
+Treat the work as complete only when:
+
+1. The project builds or typechecks.
+2. The server starts or its emulator loads the instrumentation without duplicate-provider or duplicate-app errors.
+3. One real AI request produces OpenInference spans.
+4. Storage Rules do not grant client SDK access to `agentpond/**`.
+5. The trace is visible after:
+
+```bash
+npx agentpond sync
+npx agentpond traces list --limit 10
+```
+
+Inspect the trace and confirm model, CHAIN, TOOL, input/output, parent-child, and session attributes that apply to the application. For short-lived processes, force-flush before exit. Do not shut down a reusable module-level provider after every Firebase request.
+
+## Attribution
+
+This workflow is adapted from Arize AI's MIT-licensed [arize-instrumentation skill](https://github.com/Arize-ai/arize-skills/tree/main/skills/arize-instrumentation). It replaces Arize-specific export and verification with AgentPond and Firebase Storage while retaining the analyze-then-implement workflow.

.agents/skills/agentpond-instrumentation/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "AgentPond Instrumentation"
+  short_description: "Add OpenInference tracing to Firebase apps"
+  default_prompt: "Use $agentpond-instrumentation to add OpenInference tracing to this Firebase application."

.agents/skills/agentpond-instrumentation/references/firebase.md

@@ -0,0 +1,90 @@
+# Firebase instrumentation and storage
+
+Use this reference for every Firebase instrumentation task.
+
+## Trusted runtime boundary
+
+`createFirebaseSpanExporter()` uses Firebase Admin and must run in trusted server code such as Cloud Functions for Firebase or another Node.js server with Firebase Admin credentials. Never add it to web, mobile, or other client bundles.
+
+If the repository has no trusted server runtime, stop and ask whether the user wants to add one. Do not work around the boundary with client credentials.
+
+## Default Firebase app
+
+The exporter derives the project ID and storage bucket from the default Firebase Admin app. Reuse the project's existing initialization whenever possible:
+
+```ts
+import { createFirebaseSpanExporter } from "@agentpond/firebase";
+
+const exporter = createFirebaseSpanExporter();
+```
+
+Add `initializeApp()` only when the server has no default app initialization. When a shared module genuinely needs defensive initialization, check the default app specifically:
+
+```ts
+import { getApp, initializeApp } from "firebase-admin/app";
+
+try {
+  getApp();
+} catch {
+  initializeApp();
+}
+```
+
+Do not use `getApps().length` for this decision. A named app can exist while the required default app is absent.
+
+## OpenTelemetry provider
+
+Inspect the installed OpenTelemetry version and existing provider before editing. Add the exporter to the existing provider rather than registering another global provider.
+
+When no provider exists, a typical Node SDK shape is:
+
+```ts
+import { createFirebaseSpanExporter } from "@agentpond/firebase";
+import { NodeSDK } from "@opentelemetry/sdk-node";
+
+const sdk = new NodeSDK({
+  traceExporter: createFirebaseSpanExporter(),
+  instrumentations: [
+    // Add the integration selected for the detected AI SDK or framework.
+  ],
+});
+
+sdk.start();
+```
+
+NodeSDK wraps `traceExporter` in a `BatchSpanProcessor`, so each exporter invocation can contain multiple spans and AgentPond writes one object per exported batch. When constructing a provider manually or tuning queue and batch settings, create a `BatchSpanProcessor` explicitly instead. Do not use `SimpleSpanProcessor` for normal production export because it invokes the exporter separately for every ended span.
+
+Adapt the construction to the installed SDK API and project lifecycle. Initialize the module before instrumented clients. Force-flush at a real lifecycle boundary when required so queued batches finish exporting; do not shut down a reusable Functions instance after every request.
+
+## Storage Rules review
+
+AgentPond writes trace objects below `agentpond/` in the Firebase Storage bucket. Firebase Admin access from the exporter and local CLI is trusted and bypasses Firebase client Storage Rules. Client SDKs must not be able to read, list, create, update, or delete those objects.
+
+1. Read `firebase.json` and locate every configured Storage Rules file.
+2. Review every `match` and `allow` expression that can overlap `agentpond/**`, including recursive wildcard rules.
+3. Remember that Firebase Rules are allow-only. If any overlapping allow condition evaluates to true, access is granted.
+4. A nested block such as this is not a deny override:
+
+```rules
+match /agentpond/{allPaths=**} {
+  allow read, write: if false;
+}
+```
+
+5. If no allow matches `agentpond/**`, leave the default-deny rules unchanged.
+6. If a broad allow matches the prefix, report a blocker. Narrow the broad match to application-owned prefixes or change its actual condition so AgentPond objects are excluded. Base the edit on the repository's rule structure; do not invent a universal exclusion snippet.
+7. When the emulator and Rules tests are configured, add or run tests proving representative authenticated and unauthenticated client requests cannot access `agentpond/**` while intended application paths still work.
+
+Do not declare setup complete while a broad client allow still exposes AgentPond trace data.
+
+## Firebase project selection and verification
+
+Select the project with Firebase, not AgentPond environments:
+
+```bash
+firebase use <alias-or-project-id>
+npx agentpond sync
+npx agentpond traces list --limit 10
+```
+
+Do not use `npx agentpond env init` or `npx agentpond env use` for Firebase projects.

.agents/skills/agentpond-instrumentation/references/openinference.md

@@ -0,0 +1,40 @@
+# OpenInference integration
+
+Use current official OpenInference documentation to select packages and initialization for the detected AI SDK or framework.
+
+## Routing
+
+1. Prefer a framework-native integration when it captures model, chain, and tool activity.
+2. Otherwise select the provider-specific `@arizeai/openinference-*` instrumentor matching imports actually used by the server.
+3. Do not add both framework and provider instrumentation when that would duplicate spans.
+4. If no auto-instrumentor exists, retain normal OpenTelemetry tracing and add OpenInference semantic attributes to manual spans.
+
+Common JavaScript surfaces include OpenAI, Anthropic, LangChain, Bedrock, Vercel AI SDK, MCP, and GenAI semantic-convention adapters. Verify the current package name and version before installation rather than guessing from this list.
+
+## Initialization order
+
+Set up the Firebase exporter and tracer provider, register instrumentations, and only then import or construct AI clients. Respect any framework-specific preload or bootstrap mechanism.
+
+If the application already has a global provider, add the AgentPond exporter to it. Do not replace existing exporters unless the user explicitly asks.
+
+## Manual spans
+
+Use manual spans for custom application steps that auto-instrumentation cannot see:
+
+- `CHAIN`: orchestration or agent-loop boundaries
+- `TOOL`: each tool invocation, including input, output, and error status
+- `AGENT`: a meaningful agent execution boundary when the framework does not emit one
+
+Set `openinference.span.kind` and the applicable `input.value`, `output.value`, and MIME-type attributes. Avoid recording secrets or unnecessary personal data.
+
+## Sessions
+
+Set `session.id` on the outer CHAIN or AGENT span. Generate it once at the conversation boundary and reuse it for every turn in that conversation. Auto-instrumented model and tool spans should be children of that outer span.
+
+## Flush and verification
+
+- Long-running servers: keep the provider alive and flush at supported lifecycle boundaries.
+- Short-lived scripts and test commands: force-flush and shut down before process exit.
+- Firebase request handlers: do not shut down a module-level provider after every request.
+
+Run a real application request and verify the resulting trace rather than treating compilation alone as success. Confirm span kinds, inputs/outputs, parent-child relationships, tool results, and session grouping.

.agents/skills/agentpond/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: agentpond
+description: Inspect and analyze AgentPond traces, observations, sessions, and scores with focused CLI commands and DuckDB SQL. Use when investigating agent behavior, querying trace data, comparing sessions, reviewing annotations, or diagnosing failures after traces have already been collected.
+---
+
+# AgentPond trace analytics
+
+Use AgentPond to inspect collected trace data.
+
+Read these references when relevant:
+
+- Data-access commands and environment selection: [references/cli.md](references/cli.md)
+- DuckDB tables and SQL examples: [references/duckdb-schema.md](references/duckdb-schema.md)
+- Trace investigation workflow: [references/error-analysis.md](references/error-analysis.md)
+
+## Select the data source
+
+First determine whether the current directory is inside a Firebase project.
+
+For Firebase, select the project with Firebase:
+
+```bash
+firebase use <alias-or-project-id>
+npx agentpond sync
+```
+
+AgentPond follows the Firebase CLI's active project selection, including
+selections stored globally when the project has no `.firebaserc`.
+If skill installation is cancelled, `init` stops without printing the coding-agent prompt.
+
+For non-Firebase storage, inspect and select an existing AgentPond environment:
+
+```bash
+npx agentpond env current
+npx agentpond env list
+npx agentpond env use <name>
+npx agentpond sync
+```
+
+Select only an existing non-Firebase environment as part of an analysis request.
+
+## Inspect traces
+
+Start with focused commands:
+
+```bash
+npx agentpond traces list --limit 25
+npx agentpond traces get <trace-id>
+npx agentpond observations list --traceId <trace-id>
+npx agentpond scores list --traceId <trace-id>
+```
+
+Inspect a session when behavior spans multiple traces:
+
+```bash
+npx agentpond sessions list
+npx agentpond sessions get <session-id>
+```
+
+Use SQL for joins, aggregation, time windows, raw event inspection, or cost analysis:
+
+```bash
+npx agentpond sql "select id, name, session_id, total_cost from traces order by start_time desc limit 10"
+```
+
+## Report findings
+
+Separate confirmed observations from inference. Include the trace or session IDs inspected, commands or SQL used, the observed pattern, the likely cause, and the smallest useful code, prompt, or workflow change.

.agents/skills/agentpond/references/cli.md

@@ -0,0 +1,49 @@
+# AgentPond data-access CLI
+
+Run AgentPond through `npx` unless it is installed globally.
+
+## Select data
+
+Firebase project selection is owned by Firebase:
+
+```bash
+firebase use <alias-or-project-id>
+npx agentpond sync
+```
+
+AgentPond detects the Firebase root from `.firebaserc` or `firebase.json`, follows the active selection stored by the Firebase CLI even when `.firebaserc` is absent, uses that project ID for the local cache name, reads the Firebase project data, and ignores AgentPond environment selection.
+
+`npx agentpond init` verifies that both AgentPond skills exist after installation. Cancelling the Skills CLI stops setup without printing a success message or coding-agent prompt.
+
+For non-Firebase storage, select an existing environment:
+
+```bash
+npx agentpond env current
+npx agentpond env list
+npx agentpond env use production
+npx agentpond --env staging sync
+```
+
+Sync the selected environment before querying when recent data matters.
+
+## Query commands
+
+```bash
+npx agentpond sync
+npx agentpond sync --json
+
+npx agentpond traces list --limit 25
+npx agentpond traces get <trace-id>
+npx agentpond observations list --traceId <trace-id>
+
+npx agentpond sessions list
+npx agentpond sessions get <session-id>
+
+npx agentpond scores list --traceId <trace-id>
+npx agentpond scores list --observationId <observation-id>
+
+npx agentpond sql "select * from traces limit 10"
+npx agentpond sql "select * from scores where trace_id = '<trace-id>'" --json
+```
+
+Use JSON output when another tool needs to consume the result. Use focused commands for individual resources and SQL for aggregation, joins, time filtering, raw events, and cost analysis.