Skip to content

Hot cold local dev#4899

Open
dslear wants to merge 27 commits into
masterfrom
hot-cold-local-dev
Open

Hot cold local dev#4899
dslear wants to merge 27 commits into
masterfrom
hot-cold-local-dev

Conversation

@dslear

@dslear dslear commented May 6, 2026

Copy link
Copy Markdown
Collaborator

Adding local development configurations to run the Lookout hot/cold partitioned database in parallel with the existing Lookout database. This includes the VSCode configurations in the .vscode directory, the goreman procfiles and config files under _local, and the JetBrains configurations under .run

dslear and others added 3 commits April 29, 2026 09:04
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
@dslear dslear marked this pull request as draft May 6, 2026 18:33
@greptile-apps

greptile-apps Bot commented May 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds local development configurations for running the Lookout hot/cold partitioned database stack in parallel with the existing Lookout stack, covering goreman Procfiles, JetBrains run configs, and VSCode launch/task configs.

  • goreman (hot-cold.Procfile): Adds lookouthc, lookouthcingester, and lookouthcui processes alongside the existing ones; vite.config.mts gains VITE_PORT and VITE_CACHE_DIR env-var overrides to allow two simultaneous dev servers without cache collisions.
  • JetBrains (.run/): New run configs for Lookout HC, Lookout HC Ingester, Lookout HC UI, Binoculars, and a lookouthcPostgresMigration migration config (with ARMADA_EXPERIMENTALHOTCOLDSPLIT=true); a new Armada HC compound ties them all together.
  • VSCode (.vscode/): Adds Lookout HC and Lookout HC Ingester launch configs plus an Armada HC compound; the Set up HC dependencies task sequence handles lookouthc database creation and migrations using cmd/lookout with the HC config file.

Confidence Score: 5/5

This is a developer tooling-only change with no impact on production code paths; safe to merge.

All changes are limited to local dev configuration files (Procfiles, IDE run configs, init scripts, and a Vite config). No production logic, no database migrations that ship to prod, and no changes to application code outside of the Vite config's env-var overrides. Previously reported issues (missing ARMADA_EXPERIMENTALHOTCOLDSPLIT in the migration config, print_error called before definition, wrong binary path in localdev-init.sh) are resolved in this revision.

_local/procfiles/hot-cold.Procfile — the lookouthcui process starts its dev server before the openapi generation step finishes.

Important Files Changed

Filename Overview
_local/procfiles/hot-cold.Procfile New Procfile for running hot/cold stack alongside standard stack; lookouthcui waits only for yarn install, not for openapi generation, creating a brief race window.
.run/Lookout HC Ingester.run.xml New JetBrains run config for HC ingester; field stale (points to cmd/lookout), but ignored at runtime since kind=FILE.
.run/Armada HC.run.xml New JetBrains compound config including all services plus Lookout HC stack; looks correct.
.run/lookouthcPostgresMigration.run.xml New JetBrains migration run config; correctly sets ARMADA_EXPERIMENTALHOTCOLDSPLIT=true and targets the lookouthc database.
.run/Lookout HC.run.xml New JetBrains run config for Lookout HC; correctly sets experimentalHotColdSplit, separate API port, and dedicated lookouthc database.
.run/Lookout HC UI.run.xml New JetBrains shell config for HC UI dev server; correctly sets VITE_PORT=3001, PROXY_TARGET=8091, and VITE_CACHE_DIR to avoid cache collision.
.vscode/launch.json Adds Lookout HC Ingester and Lookout HC VSCode launch configs plus an Armada HC compound; CORS origins updated for HC ports.
.vscode/tasks.json Adds Migrate Lookout HC database task using cmd/lookout with _local/lookouthc/config.yaml (experimentalHotColdSplit=true), and a Set up HC dependencies sequence task.
_local/lookouthc/config.yaml New local goreman config for HC Lookout; correct ports, experimentalHotColdSplit=true, and lookouthc database.
_local/lookouthcingester/config.yaml New goreman config for HC ingester; targets lookouthc database with distinct subscription name and experimentalHotColdSplit=true.
config/lookouthc/config.yaml New production-facing HC config; intentionally omits experimentalHotColdSplit (confirmed by team as not-yet-production feature).
scripts/localdev-init.sh Adds --hotCold flag to create lookouthc database and run HC migrations via cmd/lookout with the HC config file; helper functions defined before argument parsing.
internal/lookoutui/vite.config.mts Adds VITE_CACHE_DIR and VITE_PORT env-var overrides to allow two parallel dev server instances without cache collision.
developer/dependencies/postgres-init.sh Creates lookouthc database and grants docker privileges alongside existing lookout database setup.
internal/lookouthc/README.md New README describing the hot/cold local dev setup for goreman, JetBrains, and VSCode workflows.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    subgraph Setup["Local Dev Initialization"]
        A[localdev-init.sh --hotCold] --> B[Create lookout DB]
        A --> C[Create lookouthc DB]
        B --> D[Migrate lookout schema\ncmd/lookout --migrateDatabase]
        C --> E[Migrate lookouthc schema\ncmd/lookout --migrateDatabase\n--config _local/lookouthc/config.yaml\nexperimentalHotColdSplit: true]
    end

    subgraph StandardStack["Standard Lookout Stack"]
        F[lookoutingester\nport 9006\nDB: lookout] --> G[lookout server\nport 8089\nDB: lookout]
        G --> H[lookoutui\nVITE_PORT=3000\nPROXY→8089]
    end

    subgraph HCStack["Hot/Cold Lookout Stack"]
        I[lookouthcingester\nport 9009\nexperimentalHotColdSplit: true\nDB: lookouthc] --> J[lookouthc server\nport 8091\nexperimentalHotColdSplit: true\nDB: lookouthc]
        J --> K[lookouthcui\nVITE_PORT=3001\nPROXY→8091\nVITE_CACHE_DIR=.vite-hc]
    end

    subgraph Shared["Shared Services"]
        L[Pulsar] --> F
        L --> I
        M[Armada Server\nport 8080/8081] --> H
        M --> K
        N[Binoculars\nport 8082/8084] --> H
        N --> K
        O[PostgreSQL\nport 5432] --> F
        O --> I
    end
Loading

Reviews (16): Last reviewed commit: "remove wait for postgres" | Re-trigger Greptile

Comment thread cmd/lookouthc/main.go Outdated
Comment thread cmd/lookouthc/main.go Outdated
Comment thread .run/lookouthcPostgresMigration.run.xml
Signed-off-by: David Slear <david_slear@yahoo.com>
@dslear dslear marked this pull request as ready for review May 6, 2026 18:43
@dslear dslear marked this pull request as draft May 6, 2026 18:45
dslear added 2 commits May 6, 2026 13:46
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
@dslear dslear marked this pull request as ready for review May 6, 2026 18:50
@dslear dslear marked this pull request as draft May 6, 2026 18:50
dslear and others added 5 commits May 6, 2026 13:55
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
@dslear dslear marked this pull request as ready for review May 8, 2026 20:18
Comment thread scripts/localdev-init.sh
Comment thread scripts/localdev-init.sh Outdated
Comment thread config/lookouthc/config.yaml
Comment thread .vscode/tasks.json
dslear added 2 commits May 8, 2026 15:31
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Comment thread _local/procfiles/no-auth.Procfile Outdated
Signed-off-by: David Slear <david_slear@yahoo.com>
Comment thread _local/lookouthc/config.yaml
dslear and others added 5 commits May 26, 2026 09:18
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
dslear and others added 4 commits May 27, 2026 08:49
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Comment thread .run/Lookout UI.run.xml Outdated
Comment thread .vscode/launch.json Outdated
Comment thread .vscode/tasks.json Outdated
Comment thread internal/lookoutui/package.json Outdated
Comment thread internal/lookoutui/vite.config.mts Outdated
dslear added 4 commits May 27, 2026 12:26
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Signed-off-by: David Slear <david_slear@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants