Skip to content

Adding pyspark example to docs#10

Merged
Jday7879 merged 8 commits intomainfrom
pyspark-example
Mar 3, 2026
Merged

Adding pyspark example to docs#10
Jday7879 merged 8 commits intomainfrom
pyspark-example

Conversation

@ChrisSoderberg-ONS
Copy link
Collaborator

Proposed Changes

  • Adding a pyspark example to the docs
  • Note that the output report is not consistent between the pandas validator and pyspark validator

Related Issues

  • Related to #
  • Fixes #

Pre-requisites

This section may not be fully required if the branch is not merging into main.
Please indicate items that aren't necessary and why, with comments around incomplete checks.

  • Version number has been incremented, according to SemVer
  • Changelog has been updated, listing changes to this version. Use the keep a changelog format
  • New features are tested
  • New features are documented using the numpydoc docstring format
  • Other relevant package documentation is updated
  • For new functionality, examples are included in the docs or a feature request has
    been made for it/them
  • Required workflows and pre-commits succeed

Copy link
Collaborator

@Jday7879 Jday7879 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes make sense and example is great! I've done a minor restructure with main.py so a user can run check_and_validate without having pandera or pyspark installed. We will be revisiting this to check in future sprints

@Jday7879 Jday7879 merged commit 5e37dad into main Mar 3, 2026
6 checks passed
@Jday7879 Jday7879 deleted the pyspark-example branch March 3, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants