Feat/dbt fusion - #1657
Draft
folhesgabriel wants to merge 5 commits into
Draft
Feat/dbt fusion#1657folhesgabriel wants to merge 5 commits into
folhesgabriel wants to merge 5 commits into
Conversation
Migração faseada e reversível do engine dbt-core (Python) para o dbt Fusion (Rust), atrás de um switch de runtime (DBT_ENGINE), com dbt-core mantido como engine de produção até o adapter BigQuery do Fusion ir a GA. - dbt_project.yml: require-dbt-version >=1.8.0,<3.0.0 (aceito por ambos engines) - packages.yml: dbt_utils 1.1.1 -> 1.3.0 (compatível com Fusion) - Dockerfile: instala binário Fusion (dbtf) coexistindo com o core (dbt) - cd-staging.yaml: job fusion-checks paralelo (continue-on-error) em PRs test-dev-model - engine.py (novo): run_dbt_command() despacha por DBT_ENGINE (core=dbtRunner, fusion=subprocess dbtf) - flows.py / tasks.py: passam a usar run_dbt_command() - CONTRIBUTING.md: seção Fusion (instalação, dois engines, ELv2, adapter beta) - ONBOARDING.md: handoff da migração + plano da Fase 0 (correções para dbtf parse)
…ING.md Promove o subconjunto operacional dos docs internos (gitignored docs/dbt-fusion-*.md) para o CONTRIBUTING.md, que é a doc versionada: - gotcha do packages-install-path (/app/dbt_packages) por engine - seção 'Selecionando o engine (DBT_ENGINE)': DBT_ENGINE/DBT_FUSION_BIN/ DBT_FUSION_VERSION, onde a chave mora (engine.py), setup no worker Prefect, e rollback instantâneo - checklist expandido de macros/testes sensíveis ao engine + comandos dbtf deps/parse/compile - nota sobre o formato 'arguments:' de testes exigido pelo Fusion (exemplos mantidos no formato atual: dbt-core 1.8.8 não aceita 'arguments:')
…nsform Fusion requires generic-test args nested under `arguments:`, which dbt-core 1.8.x (the production engine) rejects. Supporting both on the same files would need dbt-core >=1.10, which requires protobuf>=5 — incompatible with the project's pinned Google libs (google-analytics-data, google-api-core, googleapis-common-protos all need protobuf<5). So the repo keeps the old test-arg format (core-compatible) and the `arguments:` conversion runs ephemerally at build time for Fusion only (CI fusion-checks) via `uvx dbt-autofix deprecations` — it is not committed. Real bug fixes below are valid on both engines (old format), committed: Name fixes (dbt1501): - set_datalake_project typo (br_bd_diretorios_data_tempo__trimestre) - custom_dictionaries -> custom_dictionary_coverage (br_ibge_censo_2022) - english_tables_dictionary_coverage -> custom_dictionary_coverage_eng x10 (world_iea_timss) Latent bugs, behavior restored (dbt1060): - test: -> tests: (br_inep_ideb x4) — dead accepted_values now run - column-level where -> per-test config (br_ms_cnes x31, br_me_cnpj x5) — restores the incremental __most_recent_*__ filter that dbt-core silently dropped - materialization -> materialized (br_denatran_frota x2) - tag -> tags (br_ibge_censo_2022) Also: br_ms_pns leading-quote descriptions cleaned so build-time autofix can parse; fusion-checks CI runs ephemeral dbt-autofix and sets DBT_PACKAGES_INSTALL_PATH; CONTRIBUTING.md and ONBOARDING.md document the build-time strategy; .gitignore adds package-lock.yml (dbtf deps artifact). Validated: `uv run dbt parse` (core 1.8.8) clean on committed files; `dbtf parse` = 0 errors after the ephemeral transform (only warnings dbt1087/1089/ 1041 remain). Behavior-change caveats to validate before any Fusion run/test: - br_inep_ideb accepted_values (ensino/anos_escolares) now execute - br_ms_cnes/br_me_cnpj not_null tests now scoped to the most-recent partition - br_denatran_frota now materializes as incremental/table as intended
…ection
- build-docker-prefect3{,-dev}.yaml: thread DBT_FUSION_VERSION build-arg
(from vars.DBT_FUSION_VERSION) so the worker image installs dbtf; the
existing work-pool image PATCH step then makes the pool run Fusion.
- deploy_flows.py: new repeatable --env KEY=VALUE flag, merged into each
flow's job_variables[env] without clobbering other keys.
- cd-prefect3-staging.yaml: deploy-flow-fusion label injects DBT_ENGINE=fusion.
- Principal dbtf commands cheat-sheet (deps/parse/compile/run/test/build/ls). - DBT_FUSION_VERSION CD source note (vars.DBT_FUSION_VERSION). - Fix stale cd-staging.yaml link -> pr-checks.yaml. (Execution-test report lives at docs/fusion-execution-test.md, untracked: docs/ is gitignored.)
folhesgabriel
marked this pull request as draft
July 14, 2026 14:13
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| # Defina DBT_FUSION_VERSION para fixar a versão; vazio instala a mais recente. | ||
| ARG DBT_FUSION_VERSION="" | ||
| ENV PATH="/usr/local/bin:/root/.local/bin:${PATH}" | ||
| RUN curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh \ |
| # Defina DBT_FUSION_VERSION para fixar a versão; vazio instala a mais recente. | ||
| ARG DBT_FUSION_VERSION="" | ||
| ENV PATH="/usr/local/bin:/root/.local/bin:${PATH}" | ||
| RUN curl -fsSL https://public.cdn.getdbt.com/fs/install/install.sh \ |
There was a problem hiding this comment.
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
28 tasks
Contributor
|
@folhesgabriel esse pull request tem conflitos 😩 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Template Pull Requests - Pipeline
Nomeação do Pull Request
A nomeação de cada Pull Request (PR) deve seguir o seguinte padrão:
Draft:
Descrição do PR:
Detalhes Técnicos:
Detalhe as mudanças mais técnicas, como ajustes na pipeline, scripts ou modelo de dados utilizado.
Se alguma parte do código precisar de alguma atenção a mais, comente na linha sinalizando para os revisores.
Teste e Validações:
Relate os testes e validações relacionado aos dados/script:
Caso haja algo relacionado aos testes que vale a pena informar:
Riscos e Mitigações:
Identifique os riscos potenciais desta mudança e como mitigar esses Riscos
Dependencias:
Revisadores: