feat(onnx): GPU execution providers (CUDA/TensorRT, fp16/int8), gated

Wires the GPU serving path behind -DSQLITE_PREDICT_ONNX_GPU
(make loadable-onnx-gpu):

- CUDA and TensorRT execution providers via proper provider-options
  (Create/Update/Release), not the previous NULL stub. TensorRT honors
  trt_fp16_enable when precision=fp16. Appending fails loud if the EP is
  not in the onnxruntime build — never a silent drop to CPU.
- fp16/int8 precision allowed only in the GPU build, and only with a
  cuda/tensorrt device (pairing fp16 with cpu/coreml is rejected rather
  than silently computing fp32).
- The provider-options symbols are in every onnxruntime C API, so the GPU
  build compiles and links against the CPU onnxruntime. CI now compile-
  checks it (make loadable-onnx-gpu); real GPU execution is validated on a
  dedicated GPU job (needs onnxruntime-gpu + a GPU runner).

Verified locally against the CPU onnxruntime: the GPU build compiles and
links clean, device=cuda/tensorrt fail loud with RUNTIME_UNAVAILABLE (no
crash), fp16+cpu is rejected, and the regular onnx suite stays 124 green.
The receipt already records device+precision, so GPU results are honestly
distinguishable from the deterministic CPU path. README/ARCHITECTURE/
CHANGELOG updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: mstrathman <matthew.strathman@gmail.com>

Author: mstrathman

.github/workflows/ci.yml

@@ -127,3 +127,5 @@ jobs:
         run: make test-onnx
       - name: ASan + LSan soak of the onnx backend
         run: make test-asan-onnx
+      - name: Compile-check the GPU build (CUDA/TensorRT wiring)
+        run: make loadable-onnx-gpu

ARCHITECTURE.md

@@ -53,10 +53,16 @@ precision), run query rows in batches, and select the execution provider
 explicitly, erasing no failure into a silent CPU fallback. Weights are pinned
 by content hash, so a receipt records exactly which bytes ran, and the
 in-context receipt anchors the training rows too, so mutating the context
-breaks replay. Both run on CPU today; the GPU execution providers are the
-next layer, validated on the gated GPU CI job. Even so, the `benchmarks/`
-numbers are why the default answer for a teacher's accuracy is usually
-distillation to a small vector student.
+breaks replay. Both run on CPU. A GPU build (`make loadable-onnx-gpu`,
+`-DSQLITE_PREDICT_ONNX_GPU`) wires the CUDA and TensorRT providers and
+fp16/int8 precision; the provider-options symbols are in every onnxruntime
+C API, so it compiles and links against the CPU onnxruntime for a CI
+compile-check, while real GPU execution is validated on a dedicated GPU job.
+The receipt records the execution provider and precision, which is what
+makes GPU results honestly distinguishable from the deterministic CPU path.
+Even so, the `benchmarks/` numbers (and the TabFM→ONNX eval in
+`benchmarks/results/tabfm-onnx.md`) are why the default answer for a
+teacher's accuracy is usually distillation to a small vector student.
 
 ## Receipts, anchoring, and replay
 

CHANGELOG.md

@@ -21,8 +21,10 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   execution-provider selection. The `vector` layout serves a self-contained
   model (a distilled student or classifier/regressor); the `in_context`
   layout serves a teacher that ingests the `train_query` rows as context
-  each call (TabFM-shaped), anchoring those rows in the receipt. The default
-  build stays zero-dependency.
+  each call (TabFM-shaped), anchoring those rows in the receipt. A GPU build
+  (`make loadable-onnx-gpu`) adds the CUDA and TensorRT execution providers
+  and fp16/int8 precision, compile-checked in CI and validated on a
+  dedicated GPU job. The default build stays zero-dependency.
 - `predict_register(model_id, config_json)` to register an external model,
   pinning its weights by content hash. `_predict_models` gains `weights_uri`
   and `io_spec`; receipts record the execution provider and precision.

Makefile

@@ -78,6 +78,17 @@ loadable-onnx: $(prefix) vendor/sqlite3ext.h sqlite-predict.h $(ONNX_OBJS)
 	  $(ONNX_CFLAGS) $(CFLAGS) $(ONNX_OBJS) -o $(TARGET_LOADABLE) \
 	  $(LDFLAGS) $(ONNX_LDFLAGS)
 
+# GPU-enabled onnx build: adds the CUDA and TensorRT execution providers
+# (fp16/int8 precision) behind -DSQLITE_PREDICT_ONNX_GPU. Running it needs
+# an onnxruntime-gpu install; the provider-options symbols are in the C API
+# of every onnxruntime build, so this compiles and links against the CPU
+# onnxruntime for a CI compile-check. Real GPU execution is validated on the
+# gated GPU job.
+loadable-onnx-gpu: $(prefix) vendor/sqlite3ext.h sqlite-predict.h $(ONNX_OBJS)
+	$(CC) -fPIC -shared -std=c99 -Wall -Wextra -Ivendor/ -I./ -O3 \
+	  $(ONNX_CFLAGS) -DSQLITE_PREDICT_ONNX_GPU $(CFLAGS) $(ONNX_OBJS) \
+	  -o $(TARGET_LOADABLE) $(LDFLAGS) $(ONNX_LDFLAGS)
+
 debug: $(prefix) vendor/sqlite3ext.h sqlite-predict.h $(OBJS)
 	$(CC) -fPIC -shared -std=c99 -Wall -Wextra -Ivendor/ -I./ -g -O0 -DSQLITE_PREDICT_DEBUG $(CFLAGS) $(OBJS) -o $(TARGET_LOADABLE) $(LDFLAGS)
 
@@ -176,5 +187,5 @@ clean:
 format:
 	clang-format -i sqlite-predict.c predict-*.c
 
-.PHONY: loadable loadable-onnx debug test test-loadable test-onnx \
-  test-asan-onnx clean format
+.PHONY: loadable loadable-onnx loadable-onnx-gpu debug test test-loadable \
+  test-onnx test-asan-onnx clean format

README.md

@@ -90,8 +90,12 @@ serves two shapes, with a cached session and batched inference:
   on each call and labels the `apply_query` rows against them, the way
   TabFM works.
 
-Both run on CPU today. Running a teacher on a GPU (CUDA/TensorRT, fp16) is
-the remaining step. The default build never links onnxruntime.
+Both run on CPU today. A GPU build (`make loadable-onnx-gpu`) adds the CUDA
+and TensorRT execution providers and fp16/int8 precision; it needs an
+onnxruntime-gpu install, is compile-checked in CI, and its GPU execution is
+validated on a dedicated GPU job. Provider selection is explicit and fails
+loud: asking for `cuda` on a build without it errors, never a silent drop to
+CPU. The default build never links onnxruntime at all.
 
 ## Installing
 
@@ -110,7 +114,9 @@ build. Requires a C99 compiler. Then, from any SQLite client:
 
 For the ONNX serving path, install onnxruntime (macOS: `brew install
 onnxruntime`; Linux: extract an [onnxruntime release][ort] and point
-`ONNXRUNTIME_PREFIX` at it) and build `make loadable-onnx`.
+`ONNXRUNTIME_PREFIX` at it) and build `make loadable-onnx`. For GPU
+execution, point `ONNXRUNTIME_PREFIX` at an onnxruntime-gpu install and
+build `make loadable-onnx-gpu`.
 
 [ort]: https://github.com/microsoft/onnxruntime/releases
 

predict-onnx.c

@@ -266,12 +266,43 @@ static int onnx_build_session(const predict0_model_row *model,
 #endif
   } else if (strcmp(device, "cuda") == 0 || strcmp(device, "tensorrt") == 0) {
 #ifdef SQLITE_PREDICT_ONNX_GPU
-    OrtStatus *ep =
-        strcmp(device, "cuda") == 0
-            ? g_ort->SessionOptionsAppendExecutionProvider_CUDA_V2(so, NULL)
-            : g_ort->SessionOptionsAppendExecutionProvider_TensorRT_V2(so,
-                                                                       NULL);
-    BUILD_CHECK(ep, PREDICT_ERR_RUNTIME_UNAVAILABLE, "append GPU EP");
+    /* Real provider-options wiring. Appending fails loud if the EP is not in
+     * this onnxruntime build (i.e. onnxruntime-gpu is required). Compiled
+     * against the CPU headers for the CI compile-check; exercised for real on
+     * the gated GPU job. */
+    int fp16 = opts->precision && strcmp(opts->precision, "fp16") == 0;
+    if (strcmp(device, "cuda") == 0) {
+      OrtCUDAProviderOptionsV2 *cu = NULL;
+      BUILD_CHECK(g_ort->CreateCUDAProviderOptions(&cu),
+                  PREDICT_ERR_RUNTIME_UNAVAILABLE, "CreateCUDAProviderOptions");
+      /* CUDA EP runs the model's own dtype; fp16 comes from an fp16 model,
+       * not an EP flag. precision is recorded in the receipt regardless. */
+      OrtStatus *ap =
+          g_ort->SessionOptionsAppendExecutionProvider_CUDA_V2(so, cu);
+      g_ort->ReleaseCUDAProviderOptions(cu);
+      BUILD_CHECK(ap, PREDICT_ERR_RUNTIME_UNAVAILABLE, "append CUDA EP");
+    } else {
+      OrtTensorRTProviderOptionsV2 *trt = NULL;
+      BUILD_CHECK(g_ort->CreateTensorRTProviderOptions(&trt),
+                  PREDICT_ERR_RUNTIME_UNAVAILABLE,
+                  "CreateTensorRTProviderOptions");
+      if (fp16) {
+        const char *keys[] = {"trt_fp16_enable"};
+        const char *vals[] = {"1"};
+        OrtStatus *up =
+            g_ort->UpdateTensorRTProviderOptions(trt, keys, vals, 1);
+        if (up) {
+          g_ort->ReleaseTensorRTProviderOptions(trt);
+          rc = onnx_fail(up, PREDICT_ERR_RUNTIME_UNAVAILABLE,
+                         "trt_fp16_enable", errmsg);
+          goto fail;
+        }
+      }
+      OrtStatus *ap =
+          g_ort->SessionOptionsAppendExecutionProvider_TensorRT_V2(so, trt);
+      g_ort->ReleaseTensorRTProviderOptions(trt);
+      BUILD_CHECK(ap, PREDICT_ERR_RUNTIME_UNAVAILABLE, "append TensorRT EP");
+    }
 #else
     /* Honest state: the GPU execution providers are validated on the gated
      * GPU CI job and compiled only into the GPU build. This CPU build does
@@ -1153,13 +1184,31 @@ int predict0_onnx_predict(sqlite3 *db, const char *model_id,
     return rc;
 
   const char *precision = opts->precision ? opts->precision : "fp32";
-  if (strcmp(precision, "fp32") != 0) {
+#ifdef SQLITE_PREDICT_ONNX_GPU
+  int prec_ok = strcmp(precision, "fp32") == 0 ||
+                strcmp(precision, "fp16") == 0 ||
+                strcmp(precision, "int8") == 0;
+#else
+  int prec_ok = strcmp(precision, "fp32") == 0;
+#endif
+  if (!prec_ok) {
     *errmsg = sqlite3_mprintf(
-        "%s: precision '%s' is not in this build (fp32 only; fp16/int8 land"
-        " with the GPU path)",
+        "%s: precision '%s' is not available in this build (fp32 only;"
+        " fp16/int8 need the GPU build, loadable-onnx-gpu)",
         PREDICT_ERR_RUNTIME_UNAVAILABLE, precision);
     return SQLITE_ERROR;
   }
+  /* fp16/int8 only take effect on a GPU device; pairing them with cpu/coreml
+   * would silently compute fp32, so reject it rather than no-op quietly. */
+  if (strcmp(precision, "fp32") != 0) {
+    const char *dev = opts->device ? opts->device : "cpu";
+    if (strcmp(dev, "cuda") != 0 && strcmp(dev, "tensorrt") != 0) {
+      *errmsg = sqlite3_mprintf(
+          "%s: precision '%s' requires a cuda or tensorrt device", PREDICT_ERR_OPTIONS,
+          precision);
+      return SQLITE_ERROR;
+    }
+  }
   if (!license_ok(model->license, opts->accept_license)) {
     *errmsg = sqlite3_mprintf(
         "%s: model license '%s' requires accept_license to match",