Merge branch 'main' into fix/datamate-stdio-local-transport

Author: saravmajestic

.opencode/skills/query-optimize/SKILL.md

@@ -7,7 +7,7 @@ description: Analyze and optimize SQL queries for better performance
 
 ## Requirements
 **Agent:** any (read-only analysis)
-**Tools used:** sql_optimize, sql_analyze, sql_explain, altimate_core_equivalence, read, glob, schema_inspect, warehouse_list
+**Tools used:** altimate_core_rewrite (with `verify_equivalence: true`), sql_analyze, sql_explain, read, glob, schema_inspect, warehouse_list
 
 Analyze SQL queries for performance issues and suggest concrete optimizations including rewritten SQL.
 
@@ -20,9 +20,9 @@ Analyze SQL queries for performance issues and suggest concrete optimizations in
 
 2. **Determine the dialect** -- Default to `snowflake`. If the user specifies a dialect (postgres, bigquery, duckdb, etc.), use that instead. Check the project for warehouse connections using `warehouse_list` if unsure.
 
-3. **Run the optimizer**:
-   - Call `sql_optimize` with the SQL, dialect, and schema context if available
-   - If the user has a warehouse connection, first call `schema_inspect` on the relevant tables to build schema context for better optimization (e.g., SELECT * expansion)
+3. **Run the verified optimizer**:
+   - If the user has a warehouse connection, first call `schema_inspect` on the relevant tables to build schema context (needed both for better rewrites — e.g. SELECT * expansion — and to verify equivalence)
+   - Call `altimate_core_rewrite` with the SQL, schema context, and **`verify_equivalence: true`**. This proposes rewrites AND proves each one returns the same results as the original in a single step. The result is partitioned into **verified-equivalent** rewrites (safe to apply) and **unverified** rewrites (review before applying), so you never recommend a rewrite that silently changes semantics.
 
 4. **Run detailed analysis**:
    - Call `sql_analyze` with the same SQL and dialect to get the full anti-pattern breakdown with recommendations
@@ -32,10 +32,10 @@ Analyze SQL queries for performance issues and suggest concrete optimizations in
    - Look for: full table scans, sort operations on large datasets, inefficient join strategies, missing partition pruning
    - Include key findings in the report under "Execution Plan Insights"
 
-6. **Verify rewrites preserve correctness**:
-   - If `sql_optimize` produced a rewritten query, call `altimate_core_equivalence` to verify the original and optimized queries produce the same result set
-   - If not equivalent, flag the difference and present both versions for the user to decide
-   - This prevents "optimization" that silently changes query semantics
+6. **Equivalence verification is built into step 3** (`verify_equivalence: true`):
+   - Present the **verified-equivalent** rewrites as safe to apply.
+   - Present **unverified** rewrites separately with their reason ("review before applying") — do not recommend applying these without manual review.
+   - If no schema was available, all rewrites come back unverified; say so and recommend supplying a schema (or a warehouse connection) to enable verification.
 
 7. **Present findings** in a structured format:
 
@@ -83,4 +83,4 @@ The user invokes this skill with SQL or a file path:
 - `/query-optimize models/staging/stg_orders.sql` -- Optimize SQL from a file
 - `/query-optimize` -- Optimize the most recently discussed SQL in the conversation
 
-Use the tools: `sql_optimize`, `sql_analyze`, `sql_explain` (execution plans), `altimate_core_equivalence` (rewrite verification), `read` (for file-based SQL), `glob` (to find SQL files), `schema_inspect` (for schema context), `warehouse_list` (to check connections).
+Use the tools: `altimate_core_rewrite` with `verify_equivalence: true` (proposes rewrites AND proves they preserve results in one step), `sql_analyze`, `sql_explain` (execution plans), `read` (for file-based SQL), `glob` (to find SQL files), `schema_inspect` (for schema context), `warehouse_list` (to check connections).

CHANGELOG.md

@@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.7] - 2026-06-10
+
+### Added
+
+- **Verified query optimization — `altimate_core_rewrite` now proves a rewrite is safe before suggesting it.** A new `verify_equivalence` mode composes the rewrite engine with the equivalence checker: a candidate rewrite is labeled **VERIFIED** only when the engine affirmatively returns `equivalent === true`; everything else (including the no-schema case) is still returned but labeled "review before applying." This is the gated core for one-click verified query optimization, so an optimization that silently changes results is never presented as safe. (#918)
+- **Per-turn tool retrieval trims the tool-definition context flood.** Three flag-gated, default-off agent-loop reliability features: a per-turn tool subset (always-on core tools + lexical top-k, never dropping a tool referenced mid-trajectory) that cuts the ~78-tool definition payload sent each turn; grammar/JSON-Schema constrained decoding for local models (vLLM/LM Studio/llama.cpp); and a pluggable pre-execution critic gate for side-effecting tools. All default off, so the existing agent path is unchanged unless explicitly enabled. (#858)
+
+### Changed
+
+- **Upgraded the SQL engine `@altimateai/altimate-core` `0.4.0` → `0.5.1` and wired its new equivalence capabilities into the dbt PR reviewer.** The reviewer now forwards the project's SQL **dialect** hint to the equivalence engine, so dialect-specific compiled warehouse SQL (e.g. Snowflake semi-structured `col:field`) parses and the comparison is decided instead of abstaining on a syntax error, and it honors the engine's new authoritative **`decidable`** flag — abstaining when the engine itself says it could not decide, rather than guessing. An empty/auto-detect dialect is coerced to "no hint" so the engine is never handed an unknown dialect. (#925, #928)
+
+### Fixed
+
+- **The dbt PR reviewer is more reliable and less noisy.** Demo-parity hardening: tighter PII-review precision, reduced safe-refactor review noise, schema YAML catalog rules now run in review, and added DuckDB data-diff end-to-end coverage. (#919)
+- **dbt review no longer misclassifies YAML files.** Schema/property YAML files are classified correctly so the right rules apply to them. (#920)
+- **The `release-v0.8.5` adversarial test gate is pinned to a constant version** so it stops breaking on every subsequent release. (#923)
+
+### Internal
+
+- **Centralized code-review dispatch on PR ready.** A gated loop dispatches a centralized OCR/Gemini review when a pull request is marked ready for review. (#914)
+
 ## [0.8.6] - 2026-06-08
 
 ### Added

bun.lock

@@ -52,3 +52,21 @@ When [LSP servers](../lsp.md) are configured, the `lsp` tool provides:
 - Go-to-definition
 - Hover information
 - Completions
+
+### Tool Retrieval
+
+With the full data-engineering toolset (~78 tools), sending every tool definition on every turn floods the context window and adds distractors that hurt the model's tool selection. **Tool retrieval** trims the exposed set per turn to a relevant subset, cutting input tokens substantially at the same task quality.
+
+It is **off by default** and enabled with an environment variable:
+
+```bash
+ALTIMATE_TOOL_RETRIEVAL=1 altimate-code run "..."
+```
+
+When enabled, each turn exposes:
+
+- an always-on **core** set of essentials that are never trimmed (`bash`, `read`, `write`, `edit`, `glob`, `grep`, `list`, `task`, `todowrite`, `skill`),
+- any tool already **referenced by an in-flight tool call** (so a mid-trajectory tool is never dropped), and
+- the highest-scoring remaining tools by a deterministic lexical match against the turn's request, up to a fixed budget.
+
+Selection is deterministic and dependency-free; small tool sets are left untouched (nothing to gain). In internal benchmarks this cut input tokens by ~50% at an identical task-resolution rate.

docs/docs/configure/tools/config.md

@@ -56,33 +56,57 @@ export async function connect(config: ConnectionConfig): Promise<Connector> {
         new Promise<any>((resolve, reject) => {
           let resolved = false
           let timeout: ReturnType<typeof setTimeout> | undefined
+          let instance: any
+          // Sentinel for an open callback that fired synchronously (before
+          // `instance` was assigned): `undefined` = not yet fired, `null` =
+          // fired with success, `Error` = fired with failure. Replayed once
+          // `instance` exists.
+          let pendingOpen: Error | null | undefined
           const opts = accessMode ? { access_mode: accessMode } : undefined
-          const instance = new duckdb.Database(
-            dbPath,
-            opts,
-            (err: Error | null) => {
-              if (resolved) { if (instance && typeof instance.close === "function") instance.close(); return }
-              resolved = true
-              if (timeout) clearTimeout(timeout)
-              if (err) {
-                const msg = err.message || String(err)
-                if (msg.toLowerCase().includes("locked") || msg.includes("SQLITE_BUSY") || msg.includes("DUCKDB_LOCKED")) {
-                  reject(new Error("DUCKDB_LOCKED"))
-                } else {
-                  reject(err)
-                }
+          const closeQuietly = () => {
+            try {
+              if (instance && typeof instance.close === "function") instance.close()
+            } catch {
+              // best-effort cleanup of a half-open handle
+            }
+          }
+          const onOpen = (err: Error | null) => {
+            if (!instance) {
+              pendingOpen = err
+              return
+            }
+            if (resolved) {
+              closeQuietly()
+              return
+            }
+            resolved = true
+            if (timeout) clearTimeout(timeout)
+            if (err) {
+              // Open failed — release the half-open handle so it doesn't leak.
+              closeQuietly()
+              const msg = err.message || String(err)
+              if (msg.toLowerCase().includes("locked") || msg.includes("SQLITE_BUSY") || msg.includes("DUCKDB_LOCKED")) {
+                reject(new Error("DUCKDB_LOCKED"))
               } else {
-                resolve(instance)
+                reject(err)
               }
-            },
-          )
-          // Bun: native callback may not fire; fall back after 2s
+            } else {
+              resolve(instance)
+            }
+          }
+          instance = opts
+            ? new duckdb.Database(dbPath, opts, onOpen)
+            : new duckdb.Database(dbPath, onOpen)
+          // Bun: native callback may not fire; fall back after 2s. Arm the timer
+          // BEFORE replaying a synchronous callback so a sync resolve/reject can
+          // actually clear it (otherwise it lingers ~2s and delays process exit).
           timeout = setTimeout(() => {
             if (!resolved) {
               resolved = true
               reject(new Error(`Timed out opening DuckDB database "${dbPath}"`))
             }
           }, 2000)
+          if (pendingOpen !== undefined) onOpen(pendingOpen)
         })
 
       try {