data/br_rf_cnpj__socios_legado - #1799
luizavboas wants to merge 3 commits into
Conversation
…râmetro de download_max_parallel de 5 para 15 (era o que estava sendo manualmente inserido nas runs), mudança no padrão de datas no registro de metadados, mudança na query de br_bd_diretorios_brasil__empresa para usar br_rf_cnpj ao invés de br_me_cnpj. Também correções em arquivos com erros do Pyrefly
📝 WalkthroughWalkthroughThe PR migrates the empresa model to RF CNPJ sources, standardizes crawler date formats, increases download concurrency, adds scheduled Prefect 3 flows, and suppresses Pyrefly diagnostics in Harvard CBDB tooling. ChangesRF CNPJ pipeline
Diagnostic suppression updates
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
pipelines/crawler/rf_cnpj/tasks.py (1)
57-57: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument
download_max_parallelinmain.Add this parameter to the Google-style
Argssection. State that it limits
concurrent download chunks and defaults to 15.As per coding guidelines, "
**/*.py: Target Python 3.10, enforce Ruff with a
79-character line length, and add Google-Style type hints and docstrings to
functions."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipelines/crawler/rf_cnpj/tasks.py` at line 57, Document the download_max_parallel parameter in main’s Google-style Args section, stating that it limits concurrent download chunks and defaults to 15; keep the description within the project’s 79-character line limit.Source: Coding guidelines
pipelines/datasets/flows.py (1)
10-29: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd type and API documentation for the flow factory.
Add a return annotation and a Google-style docstring to
_rf_cnpj_flow. Add a
Google-style docstring to the generated_flow. Document the table, schedule,
and forwarded execution parameters.As per coding guidelines, "
**/*.py: Target Python 3.10, enforce Ruff with a
79-character line length, and add Google-Style type hints and docstrings to
functions."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipelines/datasets/flows.py` around lines 10 - 29, Update the _rf_cnpj_flow factory with an explicit return annotation and Google-style docstring describing the table identifier, schedule, and returned flow. Add a Google-style docstring to the generated _flow documenting dataset/table settings and all forwarded execution parameters, including materialization, dbt aliasing, metadata updates, target, force_run, chunking, folder date, download limits, retries, parallelism, and timeout; keep formatting within the 79-character limit.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@models/br_bd_diretorios_brasil/br_bd_diretorios_brasil__empresa.sql`:
- Around line 161-162: Update the JOIN condition in the empresa query to match
the status dictionary key `t.chave` against `a.situacao_cadastral` instead of
`a.identificador_matriz_filial`.
---
Nitpick comments:
In `@pipelines/crawler/rf_cnpj/tasks.py`:
- Line 57: Document the download_max_parallel parameter in main’s Google-style
Args section, stating that it limits concurrent download chunks and defaults to
15; keep the description within the project’s 79-character line limit.
In `@pipelines/datasets/flows.py`:
- Around line 10-29: Update the _rf_cnpj_flow factory with an explicit return
annotation and Google-style docstring describing the table identifier, schedule,
and returned flow. Add a Google-style docstring to the generated _flow
documenting dataset/table settings and all forwarded execution parameters,
including materialization, dbt aliasing, metadata updates, target, force_run,
chunking, folder date, download limits, retries, parallelism, and timeout; keep
formatting within the 79-character limit.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6f038824-48ef-4785-b79f-189e15264f81
📒 Files selected for processing (12)
models/br_bd_diretorios_brasil/br_bd_diretorios_brasil__empresa.sqlmodels/br_rf_cnpj/br_rf_cnpj__socios_legado.sqlmodels/br_tse_eleicoes/code/python/diagnostics/tier2_overrides.jsonmodels/us_harvard_cbdb/code/clean.pymodels/us_harvard_cbdb/code/gen_architecture.pymodels/us_harvard_cbdb/code/gen_dbt.pymodels/us_harvard_cbdb/code/gen_metadata_payload.pymodels/us_harvard_cbdb/code/schema_spec.pymodels/us_harvard_cbdb/code/upload.pypipelines/crawler/rf_cnpj/flows.pypipelines/crawler/rf_cnpj/tasks.pypipelines/datasets/flows.py
| `basedosdados.br_rf_cnpj.dicionario` t | ||
| on a.identificador_matriz_filial = t.chave |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Join the status dictionary with situacao_cadastral.
Line 162 joins t.chave to a.identificador_matriz_filial. This makes
situacao_cadastral depend on the matrix/branch code. Join
a.situacao_cadastral to t.chave instead.
Proposed fix
inner join
`basedosdados.br_rf_cnpj.dicionario` t
- on a.identificador_matriz_filial = t.chave
+ on a.situacao_cadastral = t.chave📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| `basedosdados.br_rf_cnpj.dicionario` t | |
| on a.identificador_matriz_filial = t.chave | |
| `basedosdados.br_rf_cnpj.dicionario` t | |
| on a.situacao_cadastral = t.chave |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@models/br_bd_diretorios_brasil/br_bd_diretorios_brasil__empresa.sql` around
lines 161 - 162, Update the JOIN condition in the empresa query to match the
status dictionary key `t.chave` against `a.situacao_cadastral` instead of
`a.identificador_matriz_filial`.
1 participant
Summary by CodeRabbit
New Features
Updates
Bug Fixes