[Feature] Create an Evaluation Pipeline using RAGAS for Answer Quality Metrics

[knoxiboy]

Description
Changes to chunking, retrieval, or prompts are currently evaluated subjectively. There is no automated way to know if a change improved or degraded the assistant's performance.

Implementation

1. Create an evaluation script using the RAGAS (Retrieval Augmented Generation Assessment) framework.
2. Build a golden dataset of test PDFs and Q&A pairs.
3. Automate the calculation of key metrics: Faithfulness (hallucination detection), Answer Relevance, Context Precision, and Context Recall.
4. Add a GitHub action to run this evaluation on a subset of the data for every pull request that modifies the RAG pipeline.

Level: Critical
Affected Files: scripts/evaluate.py, .github/workflows/evaluate.yml

[knoxiboy]

/assign gssoc'26
@param20h Please assigne this issue to me.

[github-actions]

Hi @knoxiboy! 👋

Thanks for opening this issue! We are thrilled to have you contribute to PDF-Assistant-RAG for GSSoC 2026.

📌 Before you start:

• Make sure you have read our Contributing Guidelines.
• Wait for a project admin to assign this issue to you before working on it.
• Remember to branch off dev and not main.

Happy Coding! 🚀