Evaluation Pipeline for Gaussian Process Emulators with High-dimensional Dataset

Gaussian Processes (GPs) are suffering from the "curse of the dimensionality". As input or output dimension grows up, the computation becomes intractable. This project aims to explore the state-of-the-art research of dimensionality reduction in Gaussian Process emulation. In this repository, a workflow using Snakemake (a orchestrated workflow management framework) is constructed to facilitate benchmarking different Gaussian Process models on high-dimensional input/output problems with minimal efforts.

Table of Contents

• Workflow
• Datasets
• Prerequisites
• Quick Start
• Folder Structure
• Usage
  • Configuration
  • GPU Acceleration
  • SLURM Cluster Execution
• Snakemake Built-in Benchmarking Features
• Advantage of using Snakemake
  • Language-Agnostic Design
  • CI-managed Reproducible Environments
• Comparison between Nextflow and Snakemake

Workflow

The pipeline follows a 4-step workflow:

1. Data Setup: Generate synthetic data or fetch and process real-world data using specialized modules

2. Preprocessing: Standardize, split, and save data in HDF5 format (language-agnostic)

3. Model Evaluation: Train and evaluate GP models in parallel:

   High-dimensional input models:

   • ExactGP (Python/GPyTorch)
   • DKL (Python/GPyTorch)
   • RGaSP (R/RobustGaSP)
   • PCA-RGaSP (R/RobustGaSP + PCA)

   High-dimensional output models:

   • PPGaSP, PCA-PPGaSP, kPCA-PPGaSP (R/RobustGaSP)
   • AE-PPGaSP, VAE-PPGaSP (PyTorch + R/RobustGaSP)
   • BiGP, PCA-BiGP, MTGP (Python/GPyTorch)

4. Benchmark Metrics: Compare model performance and save results

Datasets

Name	Source	Problem	Description
synthetic_100d_function	Generated by 100D function	High-dim input	100D synthetic test function
tsunami_tokushima	Zenodo	High-dim input	Tsunami inundation surrogate model
environment_spill_function	Generated by Environment spill function	High-dim output	Environmental model function
acheron	Figshare + GitHub	High-dim output	Acheron rock avalanche
synthetic_landslide	Figshare + GitHub	High-dim output	Synthetic landslide simulation

Prerequisites

1. Conda Environment Manager:

   • conda or micromamba
   • conda-lock

2. Snakemake (>= 8.0)

   conda install -c conda-forge -c bioconda snakemake

   Or with pip:

   pip install snakemake

Note

If SLURM cluster execution is needed, follows the additional installation of SLURM executor plugin:

   mamba install -c conda-forge -c bioconda snakemake-executor-plugin-slurm

Or with pip:

pip install snakemake-executor-plugin-slurm

Quick Start

1. Clone repository:

   git clone https://github.com/gary8564/gpe_bench_smk.git
   cd snakemake_demo

2. Run the default pipeline (synthetic 100D function, local):

   snakemake --profile profiles/local

Folder Structure

.
├── Snakefile                    # Main workflow orchestration
├── config.yaml                  # All configurable parameters
├── pyproject.toml               # Python package definition (src/high_dim_gp)
├── rules/                       # Modular Snakemake rules
│   ├── config.smk               # Shared config variables, paths, model selection
│   ├── utils.smk                # Reusable helpers (conda_env, gpu_env, device_flag, etc.)
│   ├── data_fetching.smk        # Data fetch command generators (Zenodo/Figshare/GitHub)
│   ├── benchmark.smk            # Performance comparison and reporting
│   ├── high_dim_input.smk       # Hub: includes all high-dim input step files
│   ├── high_dim_input/          # Per-step rules
│   ├── high_dim_output.smk      # Hub: includes all high-dim output step files
│   └── high_dim_output/         # Per-step rules
├── envs/                        # Per-rule Conda environment specifications
├── scripts/                     # Implementation scripts (Python & R)
│   ├── benchmark_metrics.py
│   ├── high_dim_input/
│   └── high_dim_output/
├── src/                         # Shared Python package (high_dim_gp)
│   └── high_dim_gp/
├── profiles/                    # Snakemake execution profiles
│   ├── local/config.yaml
│   └── slurm/config.yaml

Usage

Configuration

All parameters are defined in config.yaml:

outdir: results
use_gpu: false
case_study:
  name: synthetic_100d_function # which case study to run
  problem_type: high_dim_input # high_dim_input | high_dim_output

For high-dimensional output problems, additional parameters are required:

case_study:
  name: environment_spill_function
  problem_type: high_dim_output
  qoi: cmax # quantity of interest: hmax, vmax, or cmax
  preprocessing:
    threshold: 5e-06 # zero-truncation threshold
  dim_reduction:
    n_components: 10 # PCA/kPCA components
    latent_dim: 10 # AE/VAE latent dimension

Optionally, snakemake allows overriding config values at the command line:

snakemake --profile profiles/local \
  --config case_study="{'name': 'tsunami_tokushima', 'problem_type': 'high_dim_input'}"

If you want to extend to use your own datasets, extend the datasets section in config.yaml:

datasets:
  my_new_study:
    source: "zenodo"
    description: "Description of your dataset"
    doi: "10.5281/zenodo.XXXXXXX"
    base_url: "https://zenodo.org/records/XXXXXXX"
    files:
      - "data_file1.csv"
      - "data_file2.zip"

GPU Acceleration

Set use_gpu: true in config.yaml. This selects CUDA-enabled Conda environments for GPU-capable models (ExactGP, DKL, BiGP, MTGP, AE/VAE-PPGaSP).

SLURM Cluster Execution

snakemake --profile profiles/slurm

Snakemake Built-in Benchmarking Features

Each model evaluation rule uses Snakemake's benchmark directive to automatically capture wall clock time, CPU time, and peak memory usage. After a run, TSV files are written to:

results/<case_study>/benchmarks/
├── evaluate_exactgp.tsv        # high-dim input models
├── evaluate_dkl.tsv
├── evaluate_rgasp.tsv
├── evaluate_pca_rgasp.tsv
├── evaluate_ppgasp.tsv         # high-dim output models
├── evaluate_bigp.tsv
├── evaluate_mtgp.tsv
├── ...

Advantage of using Snakemake:

1. Language-Agnostic Design

This workflow demonstrates programming language agnosticism in scientific computing pipelines by using HDF5 as cross-language scientific data format so that the data can flow through different stages which might use different OS platform / programming languages / container images.

2. CI-managed Reproducible Environments

• Per-process environment isolation: Each Snakemake process can define its own Conda environment in envs/, allowing Python/GPyTorch, R/RobustGaSP, and other model-specific dependencies to remain isolated while still being orchestrated in one benchmark pipeline. This follows the SHOWME.how isolation principle: every computational unit should carry an explicit environment specification rather than relying on a manually configured local setup.

• CI-generated lock/pin files: The repository can use CI to regenerate platform-specific explicit Conda specifications whenever envs/*.yml changes. In Snakemake, these are stored as envs/<name>.<platform>.pin.txt beside the YAML files. This makes execution more reproducible.

Comparison between Nextflow and Snakemake

Same benchmark example is used in gpe_bench_nxf repo to demonstrate and compare how snakemake and nextflow work.

Aspect	Nextflow	Snakemake
Workflow language	Groovy DSL	Python-based
Paradigm	Channel-based (push; top-down)	File-based DAG (pull; bottom-up)
Config	.config	.yaml
Modules	modules/*.nf (one process per file)	rules/high_dim_input/*.smk, rules/high_dim_output/*.smk (one rule per file)
Intermediate files	Hidden in work/ directory	Visible in output directory

Reference

1. Mölder F, Jablonski KP, Letcher B et al. Sustainable data analysis with Snakemake [version 1; peer review: 1 approved, 1 approved with reservations]. F1000Research 2021, 10:33 (https://doi.org/10.12688/f1000research.29032.1)
2. Henri E. Bal, Jennifer G. Steiner, and Andrew S. Tanenbaum. 1989. Programming languages for distributed computing systems. ACM Comput. Surv. 21, 3 (Sep. 1989), 261–322. https://doi.org/10.1145/72551.72552