[Bugfix] br_me_comex_stat - #1766
Merged
Merged
Conversation
DaviMacielCavalcante
added
the
deploy-flow
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe Comex Stat flow now tracks source and table coverage instead of source updates and materialization. The README documents ingestion, validation, migration, scheduling, and execution behavior. ChangesComex Stat coverage reconciliation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ComexStatFlow
participant SourceCoverageRegistration
participant TableCoverageSynchronization
ComexStatFlow->>SourceCoverageRegistration: register latest source coverage
SourceCoverageRegistration-->>ComexStatFlow: return source coverage
ComexStatFlow->>ComexStatFlow: continue when source is ahead or force_run is enabled
ComexStatFlow->>TableCoverageSynchronization: synchronize table coverage after metadata updates
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
@DaviMacielCavalcante esse pull request tem conflitos 😩 |
DaviMacielCavalcante
marked this pull request as ready for review
August 7, 2026 14:51
DaviMacielCavalcante
added
test-dev-model
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pipelines/datasets/br_me_comex_stat/README.md`:
- Around line 169-188: Update the migration examples in the README to use
properly paired, consistent code fences, ensuring subsequent headings render as
Markdown rather than code. In the `parameters` example, use valid Python boolean
syntax with `True` instead of `true`.
- Around line 229-233: Align the runtime documentation with the supported Python
version declared in pyproject.toml. Update the README’s venv and execution-image
guidance consistently: either remove the Python 3.11-only StrEnum dependency
from pipelines/utils/metadata/domain.py to preserve Python 3.10 support, or
document Python 3.12 as the required runtime and revise the supported-version
declaration accordingly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: bba7da02-ebd4-4961-a0d0-f2624cd9dea0
📒 Files selected for processing (2)
pipelines/datasets/br_me_comex_stat/README.mdpipelines/datasets/br_me_comex_stat/flows.py
laura-l-amaral
approved these changes
Aug 7, 2026
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Descrição do PR
O conjunto do Comex Stat estava desatualizado: NCM parado em 2026-06 e município em
2026-05, com a fonte já publicando julho. As runs terminavam
COMPLETEDtodo dia,registrando
Poll, e encerravam no gate comNão há novas atualizações na fonte original.Motivação/Contexto: o gate do
poll_source_for_update_taskcomparasource_max_date(data de cobertura,2026-07-01) com oTable.Update.latest, que érelógio — o
register_table_materialization_taskgrava ali obq.last_modified(). Comoo dado do mês M sai no começo de M+1, a cobertura fica sempre cerca de um mês atrás do
relógio, e o gate só passa quando faz mais de um mês desde a última materialização. Na
prática o conjunto ingeria mês sim, mês não.
Estado que motivou a correção:
Table.Update(relógio)ncm_exportacaoncm_importacaomunicipio_exportacaomunicipio_importacaoCom julho na fonte, a conta do NCM ficava
2026-07-01 > 2026-07-03— falso, travadoaté setembro. As tabelas de município já tinham perdido junho pelo mesmo motivo.
Detalhes Técnicos
Principais alterações na pipeline/scripts: migração das quatro tabelas para o modelo
de reconciliação de cobertura (
pipelines/utils/metadata/poll.py), o mesmo do_run_cnes:poll_source_for_update_task→register_source_coverage_task+check_source_is_ahead_of_table_taskregister_table_materialization_task+commit_source_update_task→sync_table_coverage_taskAgora os dois lados do gate são data de cobertura. O
register_source_coverage_taskfica fora do
if not force_run, então oPollé gravado em toda run.Sem loop de backfill (como o
br_me_caged) e sem guarda de fonte vazia (como o CNES):a fonte publica o CSV do ano inteiro, então qualquer run já refaz janeiro–mês corrente e
recupera mês perdido.
to_partitionsgravadata.csvcom nome fixo por partição, entãore-subir o ano sobrescreve os mesmos objetos e o
appendnão duplica.Também adiciona
pipelines/datasets/br_me_comex_stat/README.mdcom o contexto da base:as particularidades da fonte, a validação contra os totais oficiais do MDIC, o
RawDataSourceúnico compartilhado pelas quatro tabelas, e a armadilha do seed.Mudanças nos dados e no schema: nenhuma. Os modelos dbt, o schema e as partições
ficam iguais. A cobertura BD Pro continua
PartBdprocomfree_lagde 6 meses.Impacto no desempenho: o conjunto passa a materializar todo mês em vez de mês sim,
mês não. Cada run reprocessa o ano corrente, que é o comportamento que já existia.
Teste e Validações
Riscos e Mitigações
Riscos conhecidos:
Table.Updateé pré-requisito do gate novo: sem ele, o modelo novo herda ocongelamento do antigo (foi o que segurou o CNES de 2026-06-17 a 2026-08-03). Já
aplicado, mas qualquer run do código antigo o regrava como relógio — por isso os
deployments estão pausados até este PR entrar. Conferir antes de despausar:
client.get_table_update_latest(...)tem que voltar dia 1º, nunca timestamp com hora.RawDataSourceé um só para as quatro tabelas, então oRawDataSource.Updateé umponteiro compartilhado. Não afeta o gate, que compara contra o
Table.Updatede cadatabela.
Planos de rollback: reverter o commit devolve o gate antigo. O
Table.Updatevolta aser relógio na primeira materialização, sem intervenção.
Dependencias
Summary by CodeRabbit
New Features
Documentation