[Bugfix] br_cgu_beneficios_cidadao

[DaviMacielCavalcante]

Descrição do PR

As runs do novo_bolsa_familia e da garantia_safra morrem com
FileNotFoundError: Nenhum arquivo csv encontrado em /tmp/output/<tabela> logo depois de
particionar. Este PR faz cada tabela declarar no upload o formato em que ela é de fato
gravada em disco.

• Motivação/Contexto: as duas tabelas não conseguem subir mês novo desde 31/07. A
  falha acontece antes de qualquer escrita, então nada em produção foi corrompido — as
  tabelas só não avançam.

Detalhes Técnicos

• Principais alterações na pipeline/scripts: pipelines/crawler/cgu/flows.py ganha um
  mapa de source_format por tabela, consultado só por _run_cgu_beneficios_cidadao.
  _materialize_and_metadata passa a receber esse valor nos dois upload_to_gcs, em vez
  de "csv" fixo. Os outros três _run_cgu_* continuam no default "csv", que é o que
  eles gravam.

  A causa: partition_data_beneficios_cidadao grava parquet no novo_bolsa_familia, e a
  garantia_safra usa to_partitions(file_type="parquet"), mas o source_format estava
  fixo em "csv" para as três desde a migração para o Prefect 3. Isso era inofensivo até
  a fix(br_bcb_sicor): destrava flows — poll por bytes tolerante + testes dbt #1677, que acrescentou _sync_staging_schema no ramo em que a tabela de staging já
  existe — esse ramo chama dump_header(source_format=...), que exige encontrar arquivo
  no formato declarado. O bpc nunca foi afetado, grava csv de fato.

• Mudanças nos dados e no schema: nenhuma. Os arquivos enviados são os mesmos; muda só
  o formato declarado na criação e na checagem da tabela de staging. _sync_staging_schema
  compara nomes de coluna, então não altera tipo de nada.

• Impacto no desempenho: nenhum.

Teste e Validações

Teste em https://prefect3.basedosdados.org/runs/flow-run/96747598-4dad-43bd-92a6-420c02a518fe

Summary by CodeRabbit

• Bug Fixes

  • Corrected data exports for the CGU benefícios cidadania datasets.
  • novo_bolsa_familia and garantia_safra are now uploaded in Parquet format.
  • bpc continues to be uploaded in CSV format.
  • Resolved processing failures caused by incorrectly formatted Parquet uploads, improving dataset availability and reliability.

• Documentation

  • Updated deployment documentation to reflect supported source formats and processing behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c2749ba9-20e8-4bee-a5a0-bbe64f0a3cdd

📥 Commits

Reviewing files that changed from the base of the PR and between 089af07 and 4efd7ed.

📒 Files selected for processing (1)

• pipelines/crawler/cgu/flows.py

🚧 Files skipped from review as they are similar to previous changes (1)

• pipelines/crawler/cgu/flows.py

---

📝 Walkthrough

Walkthrough

The CGU benefícios cidadão flow assigns source formats per table and propagates them to staging and production GCS uploads. Flow documentation now describes source-format handling and related execution behavior.

Changes

Benefícios cidadão source format propagation

Layer / File(s)	Summary
Table format selection and upload propagation pipelines/crawler/cgu/flows.py	The flow maps novo_bolsa_familia and garantia_safra to Parquet and bpc to CSV. _materialize_and_metadata forwards the selected format to both GCS uploads.
CGU flow documentation pipelines/crawler/cgu/flows.py, pipelines/datasets/br_cgu_beneficios_cidadao/flows.py	Documentation covers source-format propagation, the Parquet-related dump_header failure, download and polling behavior, partitioning, materialization, and control parameters.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• basedosdados/pipelines#1734: Updates flow upload calls to pass the correct table-specific source_format, including Parquet handling.
• basedosdados/pipelines#1779: Adds explicit source_format propagation for another dataset’s GCS uploads.

Suggested reviewers: winzen

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title identifies this as a bugfix for the affected CGU dataset and matches the main change.
Description check	✅ Passed	The description explains the failure, technical fix, impact, and validation, but it omits explicit risks, rollback, and dependency sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch pipeline/br_cgu_beneficios_cidadao

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pipelines/crawler/cgu/flows.py`:
- Around line 60-68: The _materialize_and_metadata function needs a Google-style
docstring documenting its source_format parameter. State that source_format
controls both GCS uploads and must match the file format produced by the
partitioning step, while preserving the existing type hints and behavior.
- Line 38: Wrap the comment beginning “Formato em que cada tabela do
br_cgu_beneficios_cidadao” in flows.py so each resulting comment line is at most
79 characters, preserving its meaning.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8c628354-6603-436c-b03a-3fdd69aca664

📥 Commits

Reviewing files that changed from the base of the PR and between 3e99386 and 089af07.

📒 Files selected for processing (2)

• pipelines/crawler/cgu/flows.py
• pipelines/datasets/br_cgu_beneficios_cidadao/flows.py

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request