Skip to content

feat: implement OCR regression harness with golden dataset and CI integration#521

Merged
Cedarich merged 5 commits into
Pulsefy:mainfrom
bytebinders:main
Jun 9, 2026
Merged

feat: implement OCR regression harness with golden dataset and CI integration#521
Cedarich merged 5 commits into
Pulsefy:mainfrom
bytebinders:main

Conversation

@bytebinders

Copy link
Copy Markdown
Contributor

✅ PR Description:

What was done:

  • Modular Harness: Developed a full-featured OCR regression harness in app/ai-service/regression_harness/.
  • Golden Dataset: Created a structured repository for "golden" documents and ground truth values in regression_harness/dataset/.
  • Automated Evaluation: Implemented evaluator.py to compare actual OCR output against expected fields, supporting text normalization, error classification, and confidence tracking.
  • CLI & Reporting: Added cli.py to run suites locally, providing human-readable console summaries and machine-readable JSON reports for CI artifacts.
  • CI/CD Integration: Integrated a new GitHub Actions workflow .github/workflows/ocr-regression.yml to trigger automatically on OCR-related changes.
  • Documentation: Provided a comprehensive README.md for tool usage, adding new samples, and maintenance.

Why it was done:
To establish a reliable, low-maintenance testing infrastructure that ensures OCR extraction accuracy is preserved as the AI models, prompts, or preprocessing steps evolve.

How it was verified:

  • Verified the integrity of the data models and evaluation logic.
  • Validated the CLI's reporting capabilities (JSON and Console).
  • Verified the GitHub Actions configuration for environment dependencies (Tesseract-OCR).
  • Local sanity check performed on the directory structure and file system operations.

Summary of Work:

  1. Models: models.py defines the schema for samples and results.
  2. Evaluator: evaluator.py implements the logic for field comparison and IoU calculation.
  3. CLI: cli.py provides the interface to run tests and export results.
  4. Dataset: Established ground_truth.json with sample documents.
  5. CI/CD: Added ocr-regression.yml for automated regression testing.
  6. Documentation: Created README.md.

Required line:
Closes #464

@drips-wave

drips-wave Bot commented May 29, 2026

Copy link
Copy Markdown

@bytebinders Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich , could you drop a few more GitHub issues for me to work on? I’m ready for more 🔧🙂

@Cedarich

Copy link
Copy Markdown
Contributor

Please fix work flow

@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich the issue is missing file sampl_001.png but I have fixed it.

@Cedarich

Copy link
Copy Markdown
Contributor

Fix workflow

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

@bytebinders is attempting to deploy a commit to the Cedarich's projects Team on Vercel.

A member of the Team first needs to authorize it.

@bytebinders

Copy link
Copy Markdown
Contributor Author

fixed

@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich fixed

@Cedarich

Cedarich commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Please fix failing python test

@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich fixed

@bytebinders

Copy link
Copy Markdown
Contributor Author

Merge everything and let's finish this up.

@Cedarich Cedarich merged commit 9a7851f into Pulsefy:main Jun 9, 2026
8 of 9 checks passed
@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich even though I know I won't get any points for this, I still managed to finish it. Thanks!

@Cedarich

Cedarich commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Thank so much for the contribution, you will get points for it go to the #tickets channel on discord scroll up a bit you would see I didn't receive points for an issue button click on it and select this issue, the points would be sent to you.

@bytebinders

Copy link
Copy Markdown
Contributor Author

Thank so much for the contribution, you will get points for it go to the #tickets channel on discord scroll up a bit you would see I didn't receive points for an issue button click on it and select this issue, the points would be sent to you.

that works only in 7 days after the wave.

and I know it, I just want to deliver what I started.

@bytebinders

Copy link
Copy Markdown
Contributor Author

@Cedarich, thank you for trusting me. That alone means a lot to me and is greatly appreciated. Thanks so much!

@Cedarich

Cedarich commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Thank you so much 🙏🏿

See you next wave 🌊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OCR Accuracy Regression Harness (Golden Inputs)

2 participants