Feature: Import api to staging layer#21
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR restructures the ingestion (“sources”) layer by moving it into a new top-level source/ package with a registry-based construction model, and adds functionality to load from the Luscii API (plus optional file-based staging output) into the staging layer.
Changes:
- Introduces a new
sourcepackage (interfaces + registry) and ports existing sources (local, sftp, SQL DB) to it, adding a new Luscii API source implementation. - Adds optional staging output to local files (flat CSV + hierarchical JSON) and improves watermark persistence behavior.
- Updates configuration/staging DB initialization and CLI command flow, plus updates related SQL queries, scripts, and documentation.
Reviewed changes
Copilot reviewed 31 out of 40 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/requests/luscii.http | REST Client request examples for Luscii API. |
| test/data/pja/treatments.json | Adds PJA test fixture data. |
| test/data/pja/specialisms.json | Adds PJA test fixture data. |
| test/data/pja/snomed_ct_codes.json | Adds PJA test fixture data. |
| test/data/pja/signal_priority_types.json | Adds PJA test fixture data. |
| source/watermark.go | Improves watermark logging and ensures parent dir exists on write. |
| source/stagingfiles/writer.go | New file-writer for staging output (CSV/JSON). |
| source/sqldb/source.go | Renames/ports SQL source to sqldb, adds driver-aware DDL/DML changes and incremental upsert behavior. |
| source/source.go | New top-level source interfaces (Source, FileWriter, FileWritable). |
| source/sftp/source.go | Ports SFTP source to new source package and plumbs JSON options + file writer. |
| source/registry.go | New registry for source constructors. |
| source/luscii/source.go | New Luscii API source implementation using shared loader + watermarks + json_options. |
| source/local/source.go | Ports local source, adds json_options + file writer integration, CSV tweaks. |
| source/local/loader.go | New shared JSON loader supporting incremental upsert + child table extraction + file output. |
| scripts/start-docker.ps1 | Skips Docker startup when hix-test is not configured. |
| queries/luscii/luscii_patients.sql | Adjusts identifier FK column naming in Luscii patient conversion SQL. |
| queries/hix-test/staging/hix_bsn_mapping.sql | Updates BSN mapping query to new schema/table. |
| models/luscii/lusciimodels.go | Adds generated Luscii OpenAPI models. |
| models/luscii/client/client.go | Adds retry/backoff behavior for paging requests. |
| Makefile | Adds load target. |
| loader/loader.go | Makes inserts/DDL driver-aware and uses Rebind. |
| loader/flatten.go | Adds generic struct flattener for loader package. |
| internal/source/source.go | Removes old internal source interface (moved to source/). |
| internal/source/luscii/source.go | Removes old internal Luscii source (replaced by source/luscii). |
| go.sum | Updates module checksums after dependency/structure changes. |
| go.mod | Updates module requirements and Go version directive. |
| docs/ids_werkpakket.md | Adds work package documentation. |
| docs/fenix_architecture.md | Adds extensive architecture documentation. |
| config/sources.go | New config helpers for building/loading sources and file writer + watermark path selection. |
| config/config.go | Extends config model (staging DB types + staging files + json_options) and adds NewStagingDB. |
| config/config.example.yaml | Updates sample config to reflect new capabilities/types. |
| cmd/fenix/output/manager.go | Only archives timestamped run dirs; leaves non-run dirs intact. |
| cmd/fenix/main.go | Switches to new config/source packages, separates loading vs conversion phases, updates direct-connect logic. |
| cmd/fenix/converter/conceptmap.go | Consolidates conceptmap load logging (per dir summary). |
| .gitignore | Adds ignores for local tooling/config and config/config.yaml. |
| .gitattributes | Forces LF for .sh and .sql. |
| .claude/settings.json | Adds repo-level Claude tool permissions file. |
Comments suppressed due to low confidence (2)
config/config.example.yaml:31
- In the example config, the source type is set to "sqlserver", but the code registers the SQL DB source under type "sqldb" (see source/sqldb init). Using "sqlserver" here will make source.Build fail with "unknown source type".
source/sqldb/source.go:284 - upsertRow builds the DELETE statement using the raw pk column name (unquoted). If the pk name needs quoting (reserved word, mixed case, etc.) this will fail, and it is inconsistent with how table names are quoted elsewhere in this file.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+37
to
41
| luscii_patients_id AS resource_id, | ||
| luscii_patients_id || '_' || COALESCE(value, CAST(rowid AS TEXT)) AS id, | ||
| luscii_patients_id AS parent_id, | ||
| 'Patient.identifier' AS fhir_path, | ||
| "use", |
Comment on lines
+52
to
+55
| path := filepath.Join(w.dir, table+".csv") | ||
| _, statErr := os.Stat(path) | ||
| isNew := os.IsNotExist(statErr) | ||
|
|
Comment on lines
+1
to
+8
| { | ||
| "permissions": { | ||
| "allow": [ | ||
| "Read(//c/Users/t.hetterscheid/Repo/HipsETL/**)", | ||
| "Bash(xargs grep -l \"fenix\\\\|config.LoadAllSources\")" | ||
| ] | ||
| } | ||
| } |
Comment on lines
+70
to
+88
| for _, rec := range records { | ||
| flat := make(map[string]interface{}) | ||
| parentID := fmt.Sprintf("%v", rec[cfg.IDField]) | ||
| if cfg.IDField != "" { | ||
| updatedIDs[parentID] = true | ||
| } | ||
|
|
||
| for k, v := range rec { | ||
| childCfg, isChild := cfg.Children[k] | ||
| if isChild { | ||
| if arr, ok := asObjectArray(v); ok { | ||
| childTable := table + "_" + k | ||
| if _, exists := childBatches[childTable]; !exists { | ||
| var sub JSONFileConfig | ||
| if childCfg != nil { | ||
| sub = *childCfg | ||
| } | ||
| childBatches[childTable] = &childBatch{cfg: sub} | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.