Clovertex Data Engineering Assignment

End-to-end clinical data pipeline with ingestion, cleaning, transformation, analytics, visualization, and CI/CD.

Overview

This project implements a complete data engineering pipeline for multi-source clinical datasets, including:

• Patient records from multiple hospital systems
• Lab results
• Diagnoses (ICD-10)
• Medications
• Genomic variants

The pipeline standardizes heterogeneous data, enriches it using reference datasets, and produces analytics-ready outputs.

Architecture

The pipeline follows a medallion architecture:

• Raw Layer → original unmodified data
• Refined Layer → cleaned and standardized datasets
• Consumption Layer → analytics-ready outputs

Pipeline stages:

1. Ingestion
2. Cleaning
3. Transformation
4. Integration
5. Analytics
6. Visualization

Design Decisions

1. Relative Paths Instead of Absolute

All file paths are relative to ensure portability across environments (local, Docker, CI).

2. Data Lake Structure

Separated into raw, refined, and consumption layers to maintain traceability and reproducibility.

3. Schema Standardization

Different source formats (CSV, JSON, Parquet) are normalized early in the pipeline.

4. Partitioning Strategy

Lab data is partitioned by year to improve scalability and query performance.

5. Avoiding Data Leakage

Clinical labels (e.g., pathogenic variants) are used only for analysis, not feature engineering.

6. Robust Missing Data Handling

Pipeline tolerates missing cross-domain data (patients may not exist in all datasets).

Assumptions

• Patient IDs are consistent across datasets after cleaning
• Reference files (ICD, lab ranges, gene reference) are accurate
• Missing cross-domain records are allowed and handled gracefully

Analytics

Generated outputs:

• Patient demographics summary
• Lab statistics (mean, median, std)
• Diagnosis frequency by ICD chapter
• Variant hotspots (top genes)
• High-risk patient cohort
• Anomaly detection flags

Visualization

Generated plots:

• Age distribution histogram
• Gender distribution
• Diagnosis frequency bar chart
• Genomics scatter plot
• High-risk patient summary
• Data quality overview

All plots are saved in: datalake/consumption/plots/

Docker

The pipeline is containerized using Docker.

Run:

docker-compose up --build

This executes the full pipeline end-to-end.

Outputs are stored in a mounted volume: datalake/

CI/CD

GitHub Actions workflow includes:

• Linting using Ruff
• Docker build validation

CI runs on every push to main branch.

How to Run

Local

python run_pipeline.py

Docker

docker-compose up --build