9211 csv import perf fix source hash - #6731
Draft
ttoomey wants to merge 4 commits into
Draft
Conversation
Moves `source_hash` computation for FY2026 HMIS imports out of Ruby and into PostgreSQL `BEFORE INSERT OR UPDATE` triggers, eliminating a per-row `ActiveRecord` allocation in the import hot loop. - **CSV Import Pre-Process**: Skip the per-row `klass.new(...).calculate_source_hash` for FY2026 staging tables, where the hash is now computed in the database as the row is written. The check is hoisted out of the row loop; legacy (2022/2024) and custom files keep the Ruby path. - **Source Hash Trigger Generator**: New single source of truth that renders the canonical, GUC-independent serialization (declared HUD column order minus `ExportID`, typed per column, NULL- and delimiter-safe) and the matching trigger DDL. Column list is driven off the live table schema rather than `hmis_structure` alone, so it never references a declared-but-dropped column (e.g. FY2026 Client's retired `HispanicLatinao`). - **Trigger Functions & Migration**: 24 generated `db/functions/source_hash_hmis_2026_*` functions plus a warehouse migration attaching one `BEFORE INSERT OR UPDATE OF <hashed cols>` trigger per staging table; `UPDATE OF` is scoped to hashed columns so bookkeeping-only updates don't rehash. - **Rake Task**: `source_hash:generate_functions` regenerates the committed function files from the live schema. - **Drift Spec**: Regenerates from the live schema and diffs against the committed SQL so an unreflected schema change fails CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scratch/early work on importer hashing
tagging @eanders