AyurGenix — Intelligent Ayurvedic Agentic RAG System

An end-to-end production-ready Agentic Retrieval-Augmented Generation (RAG) system for extracting, processing, and querying classical Ayurvedic medical literature.

---

Overview

AyurGenix transforms centuries-old Ayurvedic medical texts—often available only as noisy, scanned Sanskrit-English mixed PDFs—into an intelligent, queryable knowledge system. The system provides evidence-backed, contextually grounded health recommendations strictly sourced from classical Ayurvedic literature including:

• Charaka Samhita (चरक संहिता)
• Sushruta Samhita (सुश्रुत संहिता)
• Ashtanga Hridaya (अष्टांग हृदय)
• Madhava Nidana (माधव निदान)

The Problem

Classical Ayurvedic texts face significant digitization challenges:

• Mixed Languages: Sanskrit terms embedded in English text
• OCR Noise: Scanned PDFs produce gibberish, broken words, and encoding artifacts
• Lack of Structure: Page headers, footnotes, and repeated elements contaminate content
• Domain Complexity: Requires specialized knowledge for accurate spell correction and entity recognition

The Solution

AyurGenix implements a sophisticated 6-stage pipeline:

1. Intelligent Document Ingestion — Automatically detects scanned vs. text PDFs
2. OCR + Advanced Cleaning — Denoising, deskewing, and artifact removal
3. Language Classification — Custom Char-CNN model trained on 600K+ Sanskrit/English/gibberish tokens
4. Multilingual Spell Correction — Sanskrit (IAST normalization + fuzzy matching) and English (SymSpell)
5. Semantic Chunking — Context-aware text segmentation optimized for retrieval
6. Agentic RAG System — Multi-agent architecture with query expansion, advanced reranking, and confidence scoring

The system ensures hallucination-free generation by grounding all responses in retrieved source passages with full citations (source name + page number).

---

Key Features

📄 Document Processing Pipeline

• PDF Type Detection: Automatic classification of scanned vs. text-based PDFs
• Advanced OCR: Tesseract integration with multi-language support (English + Sanskrit)
• Image Preprocessing: Denoising, adaptive thresholding, morphological operations for OCR accuracy
• Header/Footer Removal: Intelligent detection and removal of repeated page elements
• Ligature Fixing: Automatic correction of OCR ligature errors (fi → fi, fl → fl, etc.)

🧠 Natural Language Processing

• Custom Char-CNN Classifier: Deep learning model (600K+ training samples) for Sanskrit/English/gibberish detection
• Sanskrit Vocabulary: 228K+ Sanskrit terms from Cologne Digital Sanskrit Dictionaries, WHO Sanskrit Glossary
• Ayurvedic Entity Recognition: 100+ canonical Ayurvedic terms with variant normalization
• Spell Correction: Dual-mode correction with Sanskrit fuzzy matching (trigram indexing) and English normalization
• Semantic Normalization: Entity-aware text cleaning preserving domain-specific terminology

🤖 Agentic RAG System

• Query Processing: Intent classification (treatment, diet, lifestyle, symptoms, etc.) and entity extraction
• Multi-Query Expansion: LLM-powered query augmentation for comprehensive retrieval
• Advanced Reranking: Multi-factor scoring combining semantic similarity, intent alignment, source diversity, and quality
• Conversational Memory: Session-scoped context tracking with user profile extraction (doshas, conditions)
• Confidence Scoring: Multi-dimensional confidence metrics (retrieval quality, citations, entity coverage)
• LangSmith Integration: Optional RAG evaluation with groundedness, context relevance, and answer quality metrics

🔌 Production Backend (FastAPI)

• RESTful API: /chat (synchronous) and /chat/stream (Server-Sent Events)
• Session Management: Isolated conversational contexts per user session
• Health Monitoring: /health endpoint with model and Pinecone status
• CORS Support: Configurable cross-origin resource sharing
• Streaming Generation: Token-by-token response streaming for real-time UX
• Error Handling: Graceful degradation with fallback responses

🎨 Interactive Frontend (Streamlit)

• Conversational UI: Message history with user/assistant distinction
• Confidence Indicators: Visual badges (High/Medium/Low) based on retrieval quality
• Source Citations: Expandable source panels with document name, page number, relevance score, and text preview
• Example Queries: Quick-start templates for common Ayurvedic questions
• Session Reset: Clear conversation history and start fresh
• Backend Health Check: Real-time backend connectivity status

🗄️ Vector Storage

• Pinecone Integration: Serverless vector database for semantic search
• Sentence-Transformers: all-MiniLM-L6-v2 embeddings (384-dimensional)
• Metadata Storage: Document source, page numbers, and extracted entities
• Scalable Indexing: Production-ready vector storage with sub-second query latency

---

Technology Stack

Core Languages & Frameworks

Component	Technology	Purpose
Backend	Python 3.12+	Core application logic
API Framework	FastAPI	High-performance async REST API
Frontend	Streamlit	Interactive web interface
Data Processing	NumPy, Pandas	Data manipulation and analysis

Machine Learning & AI

Component	Technology	Version	Purpose
LLM Framework	HuggingFace Transformers	4.36.2	Model loading and inference
Language Models	FLAN-T5, LLaMA 3 8B	-	Question answering and generation
Embeddings	Sentence-Transformers	2.5.1	Semantic text embeddings
Embedding Model	all-MiniLM-L6-v2	-	384-dim vector representations
Deep Learning	PyTorch	2.2.0	Neural network training and inference
Quantization	BitsAndBytes	0.43.0	4-bit model quantization for efficiency
GPU Acceleration	CUDA (optional)	-	GPU-accelerated inference

Natural Language Processing

Component	Technology	Purpose
OCR Engine	Tesseract + pytesseract	Text extraction from scanned PDFs
Image Processing	OpenCV	Image preprocessing for OCR
PDF Processing	PyMuPDF, pdfplumber, pdf2image	PDF parsing and rendering
Text Processing	NLTK, spaCy	Tokenization and linguistic analysis
Sanskrit Processing	indic-transliteration, sanskrit_parser	IAST normalization and transliteration
Language Detection	langid	Language identification

Vector Database & Retrieval

Component	Technology	Purpose
Vector Database	Pinecone	Serverless semantic search
Similarity Search	Cosine similarity	Semantic retrieval
Reranking	Custom multi-factor scoring	Advanced result reranking

Data Collection & Processing

Component	Technology	Purpose
Web Scraping	BeautifulSoup, Scrapy, requests	Sanskrit term collection
Data Validation	Pydantic	Request/response schema validation
File Formats	JSON, JSONL, CSV	Data storage and interchange

Model Training & Evaluation

Component	Technology	Purpose
Deep Learning Framework	TensorFlow/Keras	Char-CNN classifier training
Model Evaluation	LangSmith	RAG evaluation and tracing
Fuzzy Matching	RapidFuzz	Sanskrit spell correction

Development & Deployment

Component	Technology	Purpose
Server	Uvicorn	ASGI server for FastAPI
Environment Management	python-dotenv	Configuration management
Version Control	Git	Source code management

Why These Technologies?

• PyTorch over TensorFlow: Better ecosystem for transformer models and HuggingFace integration
• FastAPI over Flask: Native async support, automatic OpenAPI docs, superior performance
• Pinecone: Managed vector database eliminates infrastructure overhead
• Sentence-Transformers: State-of-art embeddings with minimal computational cost
• 4-bit Quantization: Enables running LLaMA 8B on consumer GPUs (6GB+ VRAM)
• Streamlit: Rapid prototyping with production-ready UI components

---

Project Structure

AyurGenix/
├── backend/                          # FastAPI REST API server
│   ├── agentic_rag_core.py          # Core RAG implementation with multi-agent reasoning
│   ├── main.py                       # FastAPI app with routes, session management
│   ├── requirements.txt              # Backend dependencies
│   └── .env.example                  # Environment variable template
│
├── frontend/                         # Streamlit web interface
│   ├── app.py                        # Interactive chat UI with SSE streaming
│   └── requirements.txt              # Frontend dependencies
│
├── preprocessing/                    # Text cleaning and normalization
│   ├── text_cleaning.py             # OCR noise removal, header/footer detection
│   ├── integrated_cleaner.py        # Unified cleaning pipeline with Char-CNN
│   ├── sanskrit_spell_checker.py    # Trigram-indexed fuzzy matching for Sanskrit
│   └── image_cleaning.py            # Image preprocessing for OCR (denoising, thresholding)
│
├── doc_loaders/                      # PDF ingestion modules
│   ├── textloader.py                # Extract text from digital PDFs (PyMuPDF)
│   └── scanned_loader.py            # OCR for scanned PDFs (Tesseract + OpenCV)
│
├── utils/                            # Helper utilities
│   ├── pdfutils.py                  # PDF type detection (scanned vs. text)
│   └── fileutils.py                 # File I/O operations (JSONL, PDF discovery)
│
├── Sanskrit Dataset Collection/      # Data collection scripts
│   ├── cologne_terms.py             # Extract terms from Cologne Digital Sanskrit Dictionaries
│   ├── who_sansterms.py             # Scrape WHO Sanskrit medical glossary
│   ├── ayur_terms_ak.py             # Scrape Ayurveda glossary from AyurvedaKendra
│   ├── combined_sans_terms.py       # Merge Sanskrit vocabularies
│   └── krishna_sans_gloss.py        # Additional Sanskrit term extraction
│
├── resources/                        # Static vocabularies and entities
│   ├── ayurveda_terms.json          # Canonical Ayurvedic term mappings (100+ terms)
│   └── ayurveda_terms_from_ak.json  # Extended Ayurvedic vocabulary
│
├── artifacts/                        # Trained models and serialized data
│   ├── best_charcnn_model.keras     # Char-CNN classifier (Sanskrit/English/Gibberish)
│   ├── tokenizer.pkl                # Character tokenizer for Char-CNN
│   ├── label_encoder.pkl            # Label encoder for classifier
│   ├── charcnn_pipeline.pkl         # Complete inference pipeline
│   └── training_log.csv             # Model training metrics
│
├── sanskrit_terms_collection/        # Processed Sanskrit vocabularies
│   ├── final_language_dataset.csv   # Complete training dataset (600K+ tokens)
│   ├── master_sanskrit_ascii_final.csv  # Merged Sanskrit terms (228K+)
│   └── [other vocabulary files]
│
├── output/                           # Processed document outputs
│   └── ayurveda_docs_final.jsonl    # Cleaned and structured Ayurvedic texts
│
├── main.py                           # Document processing orchestrator
├── final_daraset.py                 # Dataset creation and merging script
├── char_cnn_classifier.ipynb        # Model training notebook (Char-CNN)
├── vector_embeddings.ipynb          # Vector embedding and Pinecone indexing notebook
├── requirements.txt                  # Root-level dependencies
├── .gitignore                        # Git ignore rules
└── README.md                         # Project documentation

Directory Responsibilities

backend/

FastAPI production server implementing the Agentic RAG system. Handles REST API endpoints, session-scoped conversational memory, streaming responses via Server-Sent Events, and LangSmith integration for RAG evaluation.

frontend/

Streamlit-based conversational UI providing real-time chat, confidence scoring visualization, source citation display, and backend health monitoring.

preprocessing/

Text normalization pipeline handling OCR artifacts, Devanagari script removal, ligature corrections, Sanskrit spell checking with fuzzy matching, and Ayurvedic entity preservation.

doc_loaders/

PDF ingestion modules with automatic scanned vs. text PDF detection. Text PDFs use PyMuPDF extraction; scanned PDFs undergo OpenCV preprocessing + Tesseract OCR with English/Sanskrit language models.

Sanskrit Dataset Collection/

Web scraping and data collection scripts that assembled 228K+ Sanskrit terms from Cologne Digital Sanskrit Dictionaries, WHO medical glossaries, and Ayurvedic knowledge bases.

artifacts/

Trained Char-CNN classifier (Keras) achieving high accuracy on Sanskrit/English/gibberish classification task. Used during text cleaning to preserve Sanskrit terms while removing OCR noise.

resources/

Curated Ayurvedic terminology with canonical names and variants (e.g., "Ashwagandha" → ["Aswagandha", "Withania somnifera"]) for entity recognition and spell correction.

output/

Structured JSONL dataset containing cleaned Ayurvedic texts with metadata (source, page, entities). Ready for vector embedding and Pinecone ingestion.

---

System Architecture

AyurGenix implements a modular, multi-stage architecture optimized for classical Sanskrit-English medical text processing:

graph TB
    subgraph "Document Ingestion"
        A[PDF Files] --> B{PDF Type Detection}
        B -->|Text PDF| C[PyMuPDF Extraction]
        B -->|Scanned PDF| D[Image Preprocessing]
        D --> E[Tesseract OCR eng+san]
    end
    
    subgraph "Text Cleaning Pipeline"
        C --> F[Text Cleaning]
        E --> F
        F --> G[Devanagari Removal]
        G --> H[Char-CNN Classification]
        H --> I[Sanskrit Spell Checker]
        I --> J[Entity Extraction]
        J --> K[Header/Footer Removal]
    end
    
    subgraph "Vector Storage"
        K --> L[Semantic Chunking]
        L --> M[Sentence-Transformers]
        M --> N[Pinecone Index]
    end
    
    subgraph "Agentic RAG System"
        O[User Query] --> P[Query Processor]
        P --> Q[Intent Classification]
        P --> R[Entity Extraction]
        Q --> S[Multi-Query Expansion]
        S --> T[Vector Retrieval]
        N --> T
        T --> U[Advanced Reranker]
        U --> V[LLM Generation]
        V --> W[Confidence Scoring]
        W --> X[Response + Citations]
    end
    
    subgraph "API Layer"
        X --> Y[FastAPI Backend]
        Y --> Z[Streamlit Frontend]
    end

Loading

---

How It Works

1. Document Ingestion & Preprocessing

PDF Type Detection:

• The system automatically detects whether a PDF contains extractable text or is a scanned image using pdfutils.py
• Text PDFs are processed via PyMuPDF for fast extraction
• Scanned PDFs undergo image preprocessing (denoising, adaptive thresholding) before OCR

Image Preprocessing (for scanned PDFs):

# preprocessing/image_cleaning.py
1. Convert to grayscale
2. Median blur + non-local means denoising
3. Adaptive thresholding
4. Morphological operations (opening/closing)
5. Upscaling for OCR accuracy

OCR Extraction:

• Tesseract with combined eng+san language models
• English-filtering logic to remove Sanskrit/gibberish tokens
• Preserves valid English text and recognized Sanskrit terms

2. Text Cleaning & Normalization

Multi-Stage Cleaning Pipeline:

# preprocessing/text_cleaning.py
1. Unicode normalization (NFKC)
2. Ligature correction (fi→fi, fl→fl)
3. Devanagari script removal ([\u0900-\u097F])
4. Header/footer detection via line frequency analysis
5. OCR artifact removal (broken words, spacing errors)
6. Sanskrit spell correction (trigram-indexed fuzzy matching)
7. Ayurvedic entity normalization (canonical mapping)
8. Sentence tokenization (NLTK)

Char-CNN Language Classification:

• Custom-trained model on 600K+ tokens (Sanskrit/English/Gibberish)
• Trained using Keras with character-level embeddings
• Used to preserve legitimate Sanskrit terms during cleaning
• Architecture: Embedding → Conv1D → MaxPooling → GlobalMaxPooling → Dense

3. Sanskrit Vocabulary & Spell Correction

Data Sources:

• Cologne Digital Sanskrit Dictionaries (228K+ terms)
• WHO Sanskrit Medical Glossary
• AyurvedaKendra Glossary
• Custom Ayurvedic term mappings (100+ canonical entities)

Spell Correction Strategy:

• Trigram indexing for fast candidate generation
• IAST (International Alphabet of Sanskrit Transliteration) normalization
• Fuzzy matching with RapidFuzz (edit distance)
• Entity-aware correction (preserves domain-specific terminology)

4. Vector Embedding & Indexing

Embedding Generation:

# vector_embeddings.ipynb
1. Load cleaned JSONL documents
2. Generate 384-dim embeddings via Sentence-Transformers (all-MiniLM-L6-v2)
3. Create metadata: source, page, entities
4. Batch upload to Pinecone serverless index

Pinecone Configuration:

• Index: ayurveda-rag-v2
• Dimension: 384
• Metric: Cosine similarity
• Metadata fields: text, source, page, entities

5. Agentic RAG System

Query Processing Flow:

# backend/agentic_rag_core.py - chat() method
1. Retrieve conversational memory (session-scoped)
2. Query processing:
   - Intent classification (treatment/diet/lifestyle/symptoms/causes/etc.)
   - Entity extraction (doshas, herbs, conditions)
   - Multi-query expansion (LLM-powered augmentation)
3. Vector retrieval:
   - Execute expanded queries against Pinecone
   - Aggregate results with deduplication
4. Advanced reranking:
   - Semantic similarity (50%)
   - Intent alignment (25%)
   - Source diversity (15%)
   - Text quality (10%)
5. Answer generation:
   - Intent-based system prompts
   - Context-aware generation (FLAN-T5/LLaMA 3)
   - Citation insertion
6. Confidence scoring:
   - Retrieval quality (40%)
   - Source count (20%)
   - Answer length (15%)
   - Citation count (15%)
   - Entity coverage (10%)
7. Update conversational memory

Conversational Memory:

• Session-isolated memory per user
• Stores recent messages (max 20)
• Extracts user profile (conditions, doshas mentioned)
• Provides relevant history for context-aware responses

LangSmith Integration (Optional):

• Groundedness evaluation (answer grounded in context?)
• Context relevance (retrieved passages relevant to query?)
• Answer quality (QA pair quality assessment)
• Enabled via ENABLE_RAG_EVAL=true environment variable

6. API & User Interface

FastAPI Backend:

• /chat: Synchronous endpoint returning complete response
• /chat/stream: SSE streaming for token-by-token generation
• /health: System status (model loaded, Pinecone connection)
• /sessions/{session_id}: Delete conversation history
• /stats: Active sessions and message count

Streamlit Frontend:

• Real-time conversational UI
• Confidence badges (High/Medium/Low)
• Expandable source citations with preview
• Example query templates
• Backend health monitoring
• Session management (reset conversation)

---

Core Components

1. Document Loaders

TextLoader (doc_loaders/textloader.py)

• Purpose: Extract text from digital PDFs with embedded text
• Technology: PyMuPDF (fitz)
• Process: Page-by-page text extraction → post-processing → entity extraction
• Output: Structured documents with metadata (source, page, entities)

ScannedLoader (doc_loaders/scanned_loader.py)

• Purpose: OCR text extraction from scanned/image-based PDFs
• Technology: Tesseract OCR with eng+san language models
• Process:
  • PDF → Image conversion (200 DPI)
  • Image preprocessing (denoising, thresholding)
  • OCR with English filtering
  • Bounding box analysis to remove Sanskrit/gibberish
• Output: Cleaned English text with Ayurvedic terms preserved

2. Text Preprocessing Pipeline

Text Cleaner (preprocessing/text_cleaning.py)

• Functions:
  • detect_repeated_lines(): Frequency-based header/footer detection
  • remove_devanagari_and_gibberish(): Script and noise removal
  • normalize_spelling(): Fuzzy matching against canonical Ayurvedic terms
  • semantic_normalization(): Entity-aware spell correction
  • extract_ayurveda_entities(): Named entity recognition for 100+ terms
• Key Feature: Preserves Sanskrit terms while removing OCR artifacts

Sanskrit Spell Checker (preprocessing/sanskrit_spell_checker.py)

• Architecture: Trigram-indexed fuzzy matching system
• Vocabulary: 228K+ Sanskrit terms with IAST normalization
• Methods:
  • _build_trigram_index(): Fast candidate generation
  • _edit_distance(): Levenshtein distance calculation
  • correct_word(): Single-word correction with caching
  • batch_correct(): Optimized batch processing
• Performance: Sub-millisecond lookup via trigram indexing

Integrated Cleaner (preprocessing/integrated_cleaner.py)

• Purpose: Unified cleaning pipeline with ML-based classification
• Components:
  • Char-CNN classifier integration
  • OCR artifact detection patterns
  • Sanskrit term preservation logic
  • Header/footer removal with entity awareness
• Output: Statistics tracking (OCR fixes, Sanskrit preservation, junk removal)

3. Agentic RAG Core

RobustAyurvedicRAG (backend/agentic_rag_core.py)

Core Classes:

1. ConversationalMemory

   • Session-scoped message history (max 20 messages)
   • User profile extraction (conditions, doshas)
   • Relevance-based retrieval for follow-up queries
   • Methods: add(), get_history(), get_relevant(), get_user_context()

2. QueryProcessor

   • Intent classification (10 categories: treatment, diet, lifestyle, etc.)
   • Entity extraction (doshas, herbs, conditions)
   • Multi-query expansion via LLM
   • Follow-up detection (pronouns, short queries)

3. AdvancedReranker

   • Multi-factor scoring system:
     • Semantic similarity (50%)
     • Intent alignment (25%)
     • Source diversity (15%)
     • Text quality (10%)
   • Source diversification to avoid single-text bias
   • Length-based quality heuristics

4. RobustAyurvedicRAG (Main Class)

   • LLM Support: FLAN-T5 (CPU) and LLaMA 3 8B (GPU with 4-bit quantization)
   • Generation Modes: Synchronous (generate()) and streaming (generate_stream())
   • RAG Pipeline: 6-step reasoning with tracing
   • Confidence Scoring: Multi-dimensional quality assessment
   • LangSmith Integration: Optional RAG evaluation metrics

Key Methods:

• chat(): Main conversation handler with session management
• retrieve(): Multi-query vector retrieval from Pinecone
• _generate_answer(): Intent-based answer generation with citations
• _calculate_confidence(): Multi-factor confidence scoring
• _evaluate_rag(): LangSmith-based RAG evaluation (optional)

4. FastAPI Backend

Main Application (backend/main.py)

Endpoints:

• GET /: Service information
• GET /health: System health (model status, Pinecone connection)
• POST /chat: Synchronous chat (returns full response)
• POST /chat/stream: Server-Sent Events streaming
• DELETE /sessions/{session_id}: Clear session memory
• GET /stats: Active sessions and message count

Features:

• Startup model loading (singleton pattern)
• Session management dictionary
• CORS middleware for frontend integration
• Error handling with graceful degradation
• AsyncIO integration for non-blocking I/O

Configuration:

• Environment-based model selection (FLAN-T5 for CPU, LLaMA 3 for GPU)
• LangSmith tracing toggle
• Pinecone API key management

5. Streamlit Frontend

Interactive UI (frontend/app.py)

Components:

• Custom CSS styling (gradient headers, message bubbles)
• Session state management
• Backend health monitoring
• Message history with confidence badges
• Expandable source citation panels
• Example query buttons
• Streaming toggle (experimental)

User Experience:

• Confidence indicators: 🟢 High (≥70%), 🟡 Medium (50-70%), 🔴 Low (<50%)
• Source display: Document name, page number, relevance score, text preview
• Real-time backend connectivity check
• Session reset functionality

6. Model Training Pipeline

Char-CNN Classifier (char_cnn_classifier.ipynb)

Architecture:

Input (40 chars) → Embedding(128) → Conv1D(256, kernel=5) → 
MaxPooling1D → SpatialDropout1D → Conv1D(128, kernel=3) → 
GlobalMaxPooling1D → Dense(64, ReLU) → Dropout → Dense(3, Softmax)

Training:

• Dataset: 600K+ tokens (368K English, 228K Sanskrit, 100K gibberish)
• Optimizer: AdamW with learning rate scheduling
• Callbacks: EarlyStopping, ReduceLROnPlateau, ModelCheckpoint
• Epochs: 30 with early stopping
• Batch size: 512

Output:

• Model: best_charcnn_model.keras
• Tokenizer: tokenizer.pkl
• Label encoder: label_encoder.pkl
• Training log: training_log.csv

Vector Embeddings (vector_embeddings.ipynb)

Process:

1. Load cleaned JSONL documents from output/ayurveda_docs_final.jsonl
2. Generate embeddings using Sentence-Transformers (all-MiniLM-L6-v2)
3. Prepare metadata (source, page, entities)
4. Batch upload to Pinecone (100 vectors per batch)
5. Verify indexing with sample queries

Configuration:

• Embedding dimension: 384
• Pinecone index: ayurveda-rag-v2
• Serverless deployment (AWS us-east-1)

7. Data Collection Scripts

Cologne Terms Extractor (Sanskrit Dataset Collection/cologne_terms.py)

• Extracts Sanskrit terms from Cologne Digital Sanskrit Dictionaries
• Performs SLP1 → IAST transliteration
• Generates ASCII representations for matching
• Output: 228K+ Sanskrit terms with IAST and ASCII columns

WHO Sanskrit Terms (Sanskrit Dataset Collection/who_sansterms.py)

• Scrapes WHO Sanskrit medical terminology PDF
• Normalizes IAST diacritics
• Generates ASCII fallbacks
• Output: Medical Sanskrit vocabulary

Ayurveda Glossary Scraper (Sanskrit Dataset Collection/ayur_terms_ak.py)

• Web scrapes AyurvedaKendra glossary
• Normalizes terms (spaces → underscores)
• Extracts variants from parentheses
• Output: Domain-specific Ayurvedic terms

---

Installation

Prerequisites

• Python: 3.12 or higher
• GPU (Optional but recommended): NVIDIA GPU with CUDA support for LLaMA 3 8B
• Memory: Minimum 8GB RAM (16GB+ recommended for GPU models)
• Tesseract OCR: Required for scanned PDF processing
• API Keys:
  • Pinecone API key (free tier available)
  • HuggingFace token (for model access)
  • Optional: OpenAI, Groq, LangSmith API keys

System Requirements

Component	CPU Mode	GPU Mode
Model	FLAN-T5 Small	LLaMA 3 8B (4-bit)
RAM	8GB+	16GB+
VRAM	-	6GB+
Storage	5GB	10GB

Install Tesseract OCR

Windows:

# Download installer from https://github.com/UB-Mannheim/tesseract/wiki
# Install and add to PATH

# Verify installation
tesseract --version

Linux:

sudo apt-get update
sudo apt-get install tesseract-ocr tesseract-ocr-san

macOS:

brew install tesseract tesseract-lang

Clone Repository

git clone https://github.com/yourusername/AyurGenix.git
cd AyurGenix

Setup Python Environment

Create Virtual Environment:

python -m venv venv

# Windows
venv\Scripts\activate

# Linux/macOS
source venv/bin/activate

Install Dependencies

Root Dependencies (for document processing):

pip install -r requirements.txt

Backend Dependencies:

cd backend
pip install -r requirements.txt

Frontend Dependencies:

cd frontend
pip install -r requirements.txt

GPU Setup (Optional)

If you have an NVIDIA GPU with CUDA:

# Uninstall CPU PyTorch
pip uninstall torch torchvision torchaudio

# Install GPU PyTorch (CUDA 12.1)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Verify GPU availability:

import torch
print(torch.cuda.is_available())  # Should return True
print(torch.cuda.get_device_name(0))

---

Configuration

Environment Variables

Create .env file in the backend/ directory:

cd backend
cp .env.example .env

Edit .env with your API keys:

# HuggingFace (Required)
HF_TOKEN=your_huggingface_token_here

# Pinecone (Required)
PINECONE_API_KEY=your_pinecone_api_key_here

# LangSmith (Optional - for RAG evaluation)
LANGCHAIN_TRACING_V2=false
LANGCHAIN_PROJECT=ayurgenix
LANGCHAIN_API_KEY=your_langsmith_api_key_here

# Optional LLM APIs
GROQ_API_KEY=your_groq_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

# RAG Evaluation (Optional)
ENABLE_RAG_EVAL=false

RAG Configuration

Create backend/rag_config.json:

{
  "pinecone_api_key": "your_pinecone_api_key",
  "pinecone_index": "ayurveda-rag-v2",
  "local_model": "google/flan-t5-small",
  "production_model": "meta-llama/Meta-Llama-3-8B"
}

Configuration Options:

• local_model: Used when running on CPU (FLAN-T5)
• production_model: Used when GPU is available (LLaMA 3 8B with 4-bit quantization)
• pinecone_index: Your Pinecone index name

Pinecone Setup

1. Create a free account at https://www.pinecone.io
2. Create a new index:
   • Name: ayurveda-rag-v2
   • Dimensions: 384
   • Metric: Cosine
   • Cloud: AWS
   • Region: us-east-1 (or nearest region)
3. Copy your API key to .env

HuggingFace Setup

1. Create account at https://huggingface.co
2. Generate access token: Settings → Access Tokens → New Token
3. For LLaMA 3 access, accept terms at model page
4. Add token to .env

Data Preparation (Optional)

If you want to process your own Ayurvedic PDFs:

1. Place PDF files in a directory (e.g., datadir/)
2. Update INPUT_DIR in main.py:

   INPUT_DIR = r"path/to/your/pdfs"

3. Run document processing:

   python main.py

4. Run vector embedding notebook:

   jupyter notebook vector_embeddings.ipynb

Model Artifacts

Pre-trained models are stored in artifacts/:

• best_charcnn_model.keras - Language classifier
• tokenizer.pkl - Character tokenizer
• label_encoder.pkl - Label encoder

These are automatically loaded during initialization.

---

Usage

Quick Start

1. Start Backend Server:

cd backend
uvicorn main:app --reload --host 0.0.0.0 --port 8000

2. Start Frontend (in separate terminal):

cd frontend
streamlit run app.py

3. Access Application:

• Frontend: http://localhost:8501
• Backend API: http://localhost:8000
• API Docs: http://localhost:8000/docs

Example Queries

"What is pitta dosha?"
"Treatment for anxiety in Ayurveda"
"Diet recommendations for pitta imbalance"
"What are the symptoms of vata imbalance?"
"Daily routine (dinacharya) according to Ayurveda"
"Benefits of Ashwagandha"

---

API Documentation

Base URL

http://localhost:8000

Endpoints

1. Health Check

GET /health

Response:

{
  "status": "healthy",
  "model_loaded": true,
  "active_sessions": 2,
  "model": "google/flan-t5-small",
  "index": "ayurveda-rag-v2"
}

2. Synchronous Chat

POST /chat
Content-Type: application/json

Request Body:

{
  "query": "What are the symptoms of pitta imbalance?",
  "session_id": "user-123",
  "use_memory": true
}

Response:

{
  "session_id": "user-123",
  "query": "What are the symptoms of pitta imbalance?",
  "answer": "According to classical Ayurvedic texts...",
  "sources": [
    {
      "source": "Charaka_Samhita.pdf",
      "page": 45,
      "score": 0.89,
      "text_preview": "Pitta imbalance manifests..."
    }
  ],
  "confidence": 0.82,
  "reasoning": ["Memory chars: 150", "Intent=symptoms", "Retrieved=8"],
  "intent": "symptoms",
  "entities": ["pitta"],
  "response_time_seconds": 2.4
}

3. Streaming Chat

POST /chat/stream
Content-Type: application/json

Request Body:

{
  "query": "What is Agni in Ayurveda?",
  "session_id": "user-123"
}

Response (Server-Sent Events):

data: According
data: to
data: Ayurvedic
data: texts
data: ,
data: Agni
data: ...
data: [DONE]

4. Delete Session

DELETE /sessions/{session_id}

Response:

{
  "status": "deleted"
}

5. Statistics

GET /stats

Response:

{
  "total_sessions": 5,
  "total_messages": 47
}

Error Responses

400 Bad Request:

{
  "detail": "Query cannot be empty"
}

503 Service Unavailable:

{
  "detail": "RAG not initialized"
}

---

Development

Project Setup

# Clone repository
git clone https://github.com/yourusername/AyurGenix.git
cd AyurGenix

# Install all dependencies
pip install -r requirements.txt
pip install -r backend/requirements.txt
pip install -r frontend/requirements.txt

Running Tests

Currently, the project does not have a formal test suite. Testing is performed manually through:

• Backend API testing via /docs interface
• Frontend testing through Streamlit UI
• Document processing validation with sample PDFs

Code Quality

Linting (recommended):

# Install tools
pip install black flake8 isort

# Format code
black .
isort .

# Check style
flake8 backend/ frontend/ preprocessing/

Training the Char-CNN Model

If you want to retrain the language classifier:

1. Open char_cnn_classifier.ipynb in Jupyter
2. Update dataset path in cell 3
3. Run all cells
4. Model will be saved to artifacts/best_charcnn_model.keras

Processing New Documents

# 1. Place PDFs in datadir/
# 2. Update INPUT_DIR in main.py
# 3. Run processing
python main.py

# Output: output/ayurveda_docs_final.jsonl

Creating Vector Embeddings

# 1. Ensure Pinecone index exists
# 2. Open vector_embeddings.ipynb
# 3. Update paths and run all cells
jupyter notebook vector_embeddings.ipynb

Development Workflow

1. Backend Development:

cd backend
uvicorn main:app --reload --log-level debug

2. Frontend Development:

cd frontend
streamlit run app.py --server.runOnSave true

3. Document Processing:

python main.py

Performance Optimization

CPU Mode:

• Uses FLAN-T5 Small model
• Response time: 2-5 seconds
• Memory: ~2GB

GPU Mode:

• Uses LLaMA 3 8B (4-bit quantized)
• Response time: 1-3 seconds
• VRAM: ~6GB

Monitoring

Backend Logs:

# Run with debug logging
uvicorn main:app --log-level debug

LangSmith Tracing (if enabled):

• Dashboard: https://smith.langchain.com
• View RAG evaluation metrics
• Trace query flow

---

Contributing

Contributions are welcome! This project can benefit from:

Areas for Contribution

1. Model Improvements

   • Fine-tuning language models on Ayurvedic corpus
   • Improving Sanskrit spell correction accuracy
   • Enhancing entity recognition coverage

2. Features

   • Multi-language support (Hindi, Bengali, etc.)
   • Voice interface integration
   • Mobile-responsive frontend
   • Advanced visualization of dosha analysis

3. Infrastructure

   • Docker containerization
   • CI/CD pipeline setup
   • Automated testing suite
   • Performance benchmarking

4. Documentation

   • Jupyter notebooks with examples
   • Video tutorials
   • API client libraries (Python, JavaScript)

Contribution Guidelines

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Code Standards

• Follow PEP 8 style guide for Python code
• Add docstrings for all functions and classes
• Include type hints where applicable
• Test your changes locally before submitting PR

---

License

License status: No license file was found in the repository.

This project currently does not have an explicit license. If you plan to use, modify, or distribute this code, please contact the repository owner to clarify licensing terms.

Recommendation: Consider adding an open-source license such as MIT, Apache 2.0, or GPL-3.0 to clarify usage rights.

---

Roadmap

Short-term (3-6 months)

• Docker containerization for easy deployment
• Comprehensive test suite (unit, integration, E2E)
• Performance benchmarking and optimization
• Support for additional Ayurvedic texts
• Enhanced citation formatting

Medium-term (6-12 months)

• Multi-language UI (Hindi, Bengali, Tamil)
• Voice query support
• Mobile application (React Native/Flutter)
• User authentication and personalization
• Doshas assessment quiz integration

Long-term (12+ months)

• Fine-tuned domain-specific LLM
• Integration with healthcare providers
• Clinical validation studies
• Academic publication of methodology
• Community-driven knowledge base expansion

---

Acknowledgments

Data Sources

• Cologne Digital Sanskrit Dictionaries - Comprehensive Sanskrit vocabulary
• WHO Sanskrit Medical Glossary - Standardized medical terminology
• AyurvedaKendra - Ayurvedic glossary and terminology
• Classical Texts: Charaka Samhita, Sushruta Samhita, Ashtanga Hridaya, Madhava Nidana

Open Source Libraries

This project builds upon excellent open-source work:

• HuggingFace Transformers - LLM inference
• LangChain & LangSmith - RAG framework and evaluation
• Pinecone - Vector database infrastructure
• FastAPI - Modern Python web framework
• Streamlit - Rapid UI development
• PyTorch - Deep learning framework
• Tesseract OCR - Text extraction engine

---

Engineering Summary

Technical Strengths

This project demonstrates production-grade engineering practices across multiple domains:

1. Full-Stack ML System Design

• End-to-end pipeline from raw scanned PDFs to conversational AI
• Modular architecture enabling independent component testing and scaling
• Clear separation of concerns (ingestion → processing → storage → retrieval → generation)

2. Domain-Specific NLP Expertise

• Custom Char-CNN classifier achieving high accuracy on trilingual classification task
• Sophisticated Sanskrit spell correction using trigram indexing and fuzzy matching
• Entity-aware text normalization preserving technical terminology
• 228K+ Sanskrit vocabulary assembled from authoritative sources

3. Advanced RAG Implementation

• Multi-agent architecture with intent classification and query expansion
• Multi-factor reranking (semantic + intent + diversity + quality)
• Session-scoped conversational memory with profile extraction
• Confidence scoring across 5 dimensions for transparency

4. Production-Ready Backend

• FastAPI with async support and streaming responses (SSE)
• Session management for isolated user contexts
• Graceful degradation and error handling
• Optional LangSmith integration for RAG evaluation

5. OCR and Document Processing

• Automatic detection of scanned vs. text PDFs
• Advanced image preprocessing (denoising, thresholding, morphological operations)
• Intelligent header/footer removal using frequency analysis
• Bilingual OCR (English + Sanskrit) with gibberish filtering

6. Model Optimization

• 4-bit quantization enabling LLaMA 8B on consumer GPUs
• CPU/GPU mode detection with automatic model selection
• Efficient vector search with Pinecone serverless
• Caching strategies for spell correction and embeddings

Portfolio Value

For Recruiters and Hiring Managers:

This project showcases:

• ML Engineering: Custom model training, deployment, and optimization
• Backend Development: REST API design, async programming, streaming
• NLP Expertise: Text processing, OCR, multilingual systems, entity recognition
• System Architecture: Modular design, scalability, production readiness
• Domain Knowledge: Ability to work with complex, specialized datasets
• Full-Stack Skills: Backend (FastAPI) + Frontend (Streamlit) + ML pipeline
• Data Engineering: Web scraping, dataset creation, cleaning, validation
• Research Skills: Literature review, domain research, methodology design

Technical Complexity Indicators:

• 600K+ training samples for Char-CNN
• 228K+ Sanskrit vocabulary terms
• Multi-stage text cleaning pipeline
• Agentic RAG with 6-step reasoning
• Session-scoped conversational memory
• Multi-factor reranking algorithm
• Streaming generation with SSE
• Bilingual OCR with intelligent filtering

Real-World Applications

This architecture is production-ready and can be adapted for:

• Healthcare: Medical literature retrieval systems
• Legal: Case law and statutory research
• Finance: Regulatory compliance documentation
• Education: Personalized learning assistants
• Research: Academic paper search and synthesis

Engineering Maturity

Code Quality:

• Type hints and docstrings throughout
• Modular, reusable components
• Configuration management via environment variables
• Comprehensive error handling

Scalability:

• Serverless vector database (Pinecone)
• Batch processing support
• Async I/O for concurrent requests
• Efficient caching strategies

Observability:

• Health check endpoints
• Session statistics
• Training logs and metrics
• Optional LangSmith tracing

---

Disclaimer

For Educational and Research Purposes Only

This system is designed as a research and educational tool for exploring Ayurvedic literature. It is not intended to provide medical advice, diagnosis, or treatment.

• Always consult qualified healthcare professionals for medical decisions
• Do not use this system as a substitute for professional medical advice
• Ayurvedic recommendations should be implemented under expert supervision
• The system may contain errors or incomplete information

The developers and contributors assume no liability for any consequences arising from the use of this system.

---

Contact & Support

• Issues: Report bugs or request features via GitHub Issues
• Discussions: Join conversations in GitHub Discussions
• Email: [Your contact email if applicable]

---

Citation

If you use this project in your research or build upon it, please cite:

@software{ayurgenix2024,
  title={AyurGenix: Intelligent Ayurvedic Agentic RAG System},
  author={[Your Name]},
  year={2024},
  url={https://github.com/yourusername/AyurGenix}
}

---

Built with 🌿 for preserving and democratizing ancient Ayurvedic wisdom

⬆ Back to Top