Repository navigation
Parquet export does not close its writer after a batch or metadata failure #8743
Description
The public Parquet export writer closes pyarrow.parquet.ParquetWriter only on successful completion. If query_table, write_table, or metadata persistence raises, _write leaves the writer unclosed while write() closes the outer file handle.
This is distinct from #8558, which changes builder/ArrowWriter finalization: the affected path here is src/datasets/io/parquet.py::ParquetDatasetWriter._write, used by Dataset.to_parquet.
Proposed fix: put writer use in try/finally and always close it. Tests inject each failure, verify that the original exception propagates, and verify close-on-success as a control.
AI disclosure: autonomously investigated using Perplexity Computer (GPT-6.1); no human review is claimed.
Metadata
Metadata
Assignees
Labels
No labels
Activity
wpyaoxueros commentedon Oct 21, 2026
#self-assign
wpyaoxueros commentedon Oct 21, 2026
I self-assigned this issue before I detected the existing matching PRs. I have since confirmed that #8744 and #8775 both cover the same writer-cleanup path, so I will not submit duplicate work. GitHub will not let me remove my own assignment with my current permissions; could a maintainer please unassign me? Sorry for the extra noise.
wpyaoxueros commentedon Oct 23, 2026
Following up on my earlier withdrawal: I have now submitted a context-manager alternative in #8885 with real-writer regression tests for path and borrowed-buffer exports. #8744 and #8775 already cover this issue; this PR is offered as an alternative for maintainer consideration. Codex assisted with the implementation, validation and this message.