Skip to content

README.md

Latest commit

 

History

History
365 lines (271 loc) · 12.6 KB

README.md

File metadata and controls

365 lines (271 loc) · 12.6 KB

Hippocrates' Feather

An AI-powered medical form filling system for the IBM Z Datathon

Overview

Hippocrates' Feather is an innovative AI-assisted clinical forms system designed to help medical professionals efficiently manage patient appointments. The system uses live transcription and AI-powered entity extraction to automatically fill medical forms during patient-doctor consultations, reducing administrative burden and allowing doctors to focus on patient care.

Core Concept

The project addresses a critical challenge in modern healthcare: the time-consuming task of manually filling out medical forms during or after patient consultations. Hippocrates' Feather solves this by:

  1. Live Transcription: Capturing real-time audio from patient-doctor appointments using browser-based WebRTC technology
  2. AI-Powered Extraction: Automatically extracting medical entities (symptoms, medications, patient information) from the transcribed conversation
  3. Intelligent Form Filling: Populating structured clinical forms with extracted information in real-time
  4. Seamless Integration: Providing a complete patient management system with form archives and history

Project Architecture

The system consists of three main components:

1. Frontend Application (frontend/)

A Streamlit-based web application that provides the user interface for doctors:

  • Live Audio Recording: Browser-based audio capture using WebRTC
  • Real-time Transcription: Live display of patient-doctor conversation
  • AI Entity Extraction: Automatic detection of symptoms and medications from conversation
  • Dynamic Forms: Real-time form updates during consultation
  • Patient Management: Complete patient database with search and filtering
  • Form Archive: Historical form access and management

Key Technologies:

  • Streamlit for rapid UI development
  • streamlit-webrtc for browser-based audio capture
  • Pydantic for data validation
  • Python-dotenv for configuration

2. Backend API Gateway (API_Gateway/)

A FastAPI-based REST API that handles data persistence and business logic:

  • Patient Management: Full CRUD operations for patient records
  • Doctor Management: Full CRUD operations for doctor records
  • Form Submission: Batch submission of complete forms with symptoms and medications
  • Database Integration: PostgreSQL database with async SQLAlchemy ORM

Key Technologies:

  • FastAPI for REST API
  • SQLAlchemy for database ORM
  • PostgreSQL for data persistence
  • Pydantic for request/response validation

3. AI Agent (agent/)

An intelligent agent that processes transcribed text and extracts medical information:

  • Intent Detection: Identifies different types of information in conversation (PII, medications, symptoms)
  • Entity Extraction: Extracts structured data from unstructured text
  • LLM Integration: Uses OpenAI's API for intelligent extraction
  • Form Building: Constructs structured patient forms from extracted entities
  • Recommendations: Provides medical recommendations based on form data

Key Technologies:

  • OpenAI API for LLM-powered extraction
  • Instructor library for structured LLM outputs
  • Custom prompt engineering for medical domain

Features

For Medical Professionals

  • 🎙️ Live Audio Recording: Start recording directly in the browser during consultations
  • 📝 Real-time Transcription: See the conversation transcribed as it happens
  • 🤖 AI Autofill: Automatically extract symptoms and medications from conversation
  • 📋 Structured Forms: Organised clinical forms with symptoms, medications, and notes
  • 👥 Patient Database: Manage patient information and history
  • 📊 Form Archive: Access and review historical consultations
  • ✅ Quick Review: Review and edit AI-extracted information before submission

Technical Features

  • Mock Mode: Development mode with in-memory storage for testing
  • Backend Integration: Full integration with FastAPI backend
  • Error Handling: Comprehensive error handling for network and API errors
  • Data Transformation: Automatic conversion between frontend and backend data models
  • Session Management: State management for multi-page workflows
  • Responsive Design: Works on desktop and tablet devices

Project Structure

IBMZ-Datathon/
├── frontend/                    # Streamlit frontend application
│   ├── app.py                   # Main entry point
│   ├── pages/                   # Multi-page application
│   │   ├── 1_📋_Main_Dashboard.py
│   │   ├── 2_👤_Patient_Page.py
│   │   ├── 3_📝_New_Form.py
│   │   └── 4_🎙️_Live_Form.py
│   ├── components/              # Reusable UI components
│   ├── services/                # Business logic and API clients
│   ├── models/                  # Data models (Pydantic)
│   └── utils/                   # Utilities and configuration
├── API_Gateway/                 # FastAPI backend
│   └── api_gateway.py           # Main API server
├── agent/                       # AI agent for entity extraction
│   ├── form_agent/              # Form processing agent
│   ├── api_gateway.py           # Agent API endpoints
│   └── schemas.py               # Data schemas
├── db/                          # Database migrations and schemas
├── tests/                       # Test suite
├── pyproject.toml               # Python project configuration
├── requirements.txt             # Python dependencies
└── README.md                    # This file

Quick Start

Prerequisites

  • Python 3.13+
  • uv package manager (recommended) or pip
  • PostgreSQL database (for backend)
  • Modern web browser with microphone access
  • OpenAI API key (for AI agent)

Installation

Option 1: Using uv (Recommended)

  1. Install uv (if not already installed):
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
  1. Clone the repository:
git clone <repository-url>
cd IBMZ-Datathon
  1. Install dependencies:
uv sync
  1. Set up environment variables:
cp .env.example .env
# Edit .env with your configuration
  1. Run the frontend application:
uv run streamlit run frontend/app.py --server.port 8501 --server.address 0.0.0.0

Option 2: Using pip

  1. Create virtual environment:
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
  1. Install dependencies:
pip install -e .
  1. Run the application:
streamlit run frontend/app.py --server.port 8501 --server.address 0.0.0.0

Access the Application

Open your web browser and navigate to:

http://localhost:8501

Configuration

The application can be configured using environment variables (see .env.example):

Frontend Configuration

  • BACKEND_API_URL: Backend API service URL (default: https://ibm-datathon-api-gateway.onrender.com)
  • DEFAULT_DOCTOR_ID / DEFAULT_DOCTOR_NAME: Default doctor for demo
  • ASR_MODEL: Automatic Speech Recognition model (backend-controlled)
  • AI_AUTOFILL_ENABLED: Enable AI autofill from microphone
  • MOCK_API: Use mock API instead of real backend (default: false)
  • HTTP_TIMEOUT: HTTP request timeout in seconds (default: 30)
  • HTTP_RETRY_ATTEMPTS: Number of retry attempts for failed requests (default: 3)

Backend Configuration

  • DATABASE_URL: PostgreSQL connection string
  • OPENAI_API_KEY: OpenAI API key for AI agent

Usage Guide

1. Main Dashboard

  • View all patients in a searchable grid
  • Search patients by name or email
  • Create new patients
  • Navigate to patient details

2. Patient Page

  • View patient personal information
  • Access form archive with all historical forms
  • Create new clinical forms
  • Manage existing forms

3. New Form

  • Configure AI autofill settings
  • Add pre-session observations
  • Start live transcription session

4. Live Form Session

  • Audio Controls: Start/Pause/Stop/Reset recording
  • Live Transcript: Real-time conversation display
  • Form Tabs:
    • Symptoms: Add and manage patient symptoms
    • Medications: Add and manage medications
    • Summary: Review and finalise the form
  • AI Integration: Automatic entity extraction from conversation
  • Submit: Finalise and submit the form to the backend

Backend Integration

The frontend is fully integrated with the FastAPI backend. The integration handles:

  • Patient Management: Full CRUD operations for patients
  • Doctor Management: Full CRUD operations for doctors
  • Form Submission: Batch submission of complete forms with symptoms and medications
  • Error Handling: Comprehensive error handling for network and API errors
  • Data Transformation: Automatic conversion between frontend and backend data models

Backend API Endpoints

Patient Management

  • GET /patients/ - List all patients
  • POST /patients/ - Create new patient
  • GET /patients/{patient_id} - Get patient details
  • PATCH /patients/{patient_id} - Update patient
  • DELETE /patients/{patient_id} - Delete patient

Doctor Management

  • GET /doctors/ - List all doctors
  • POST /doctors/ - Create new doctor
  • GET /doctors/{doctor_id} - Get doctor details
  • PATCH /doctors/{doctor_id} - Update doctor
  • DELETE /doctors/{doctor_id} - Delete doctor

Form Submission

  • POST /forms/ - Submit complete form with symptoms and medications

Development

Tech Stack

Frontend:

  • Streamlit: Web framework for rapid UI development
  • streamlit-webrtc: Browser-based audio capture
  • Pydantic: Data validation and serialisation
  • Python-dotenv: Environment configuration

Backend:

  • FastAPI: Modern, fast web framework
  • SQLAlchemy: Database ORM
  • PostgreSQL: Relational database
  • Pydantic: Request/response validation

AI Agent:

  • OpenAI API: Large language model integration
  • Instructor: Structured LLM outputs
  • Custom prompt engineering

Running in Development Mode

  1. Enable debug mode:
export DEBUG=true
  1. Use mock API (for frontend-only development):
export MOCK_API=true
  1. Run with auto-reload:
streamlit run frontend/app.py --server.runOnSave true

Testing

The application includes comprehensive testing capabilities:

  • Mock Mode: Set MOCK_API=true to use in-memory mock data for development
  • Backend Integration: Set MOCK_API=false to test with real backend API
  • Error Scenarios: Test network failures, validation errors, and API errors
  • Data Validation: Verify field type conversions and data transformations

Troubleshooting

Common Issues

  1. Audio not working: Ensure browser microphone permissions are granted
  2. Import errors: Make sure all dependencies are installed (uv sync or pip install -e .)
  3. Port conflicts: Change the port with --server.port 8502
  4. Backend connection: Check BACKEND_API_URL in your .env file
  5. API errors: Check network connectivity and backend service status
  6. Form submission fails: Ensure at least one symptom or medication is added
  7. Data not saving: Check if MOCK_API is set correctly for your use case

Debug Mode

Enable debug logging:

export DEBUG=true
streamlit run frontend/app.py --logger.level debug

Project Goals

Hippocrates' Feather was developed for the IBM Z Datathon with the following objectives:

  1. Reduce Administrative Burden: Automate the time-consuming task of filling medical forms
  2. Improve Accuracy: Reduce human error in form transcription
  3. Enhance Patient Care: Allow doctors to focus on patients rather than paperwork
  4. Real-time Processing: Provide immediate feedback during consultations
  5. Scalable Solution: Build a system that can handle multiple consultations simultaneously

Future Enhancements

Potential improvements and extensions:

  • Multi-language support for international use
  • Integration with Electronic Health Records (EHR) systems
  • Advanced AI models for better entity extraction
  • Voice recognition for speaker identification
  • Mobile app for on-the-go consultations
  • Analytics dashboard for medical insights
  • Export functionality for forms (PDF, JSON, etc.)

License

This project is part of the IBM Z Datathon. All rights reserved.

Contributors

Developed for the IBM Z Datathon by the Hippocrates' Feather team.

Note: This project is a demonstration system for the IBM Z Datathon. For production use, additional security measures, authentication, and compliance with healthcare regulations (such as HIPAA) would be required.