FZ - Parametric Scientific Computing Framework

A powerful Python package for parametric simulations and computational experiments. FZ wraps your simulation codes to automatically run parametric studies, manage input/output files, handle parallel execution, and collect results in structured DataFrames.

Features

Core Capabilities

🔄 Parametric Studies: Automatically generate and run all combinations of parameter values (Cartesian product)
⚡ Parallel Execution: Run multiple cases concurrently across multiple calculators with automatic load balancing
💾 Smart Caching: Reuse previous calculation results based on input file hashes to avoid redundant computations
🔁 Retry Mechanism: Automatically retry failed calculations with alternative calculators
🌐 Remote Execution: Execute calculations on remote servers via SSH with automatic file transfer
📊 DataFrame Output: Results returned as pandas DataFrames with automatic type casting and variable extraction
🛑 Interrupt Handling: Gracefully stop long-running calculations with Ctrl+C while preserving partial results
🔍 Formula Evaluation: Support for calculated parameters using Python or R expressions
📁 Directory Management: Automatic organization of inputs, outputs, and logs for each case

Four Core Functions

fzi - Parse Input files to identify variables
fzc - Compile input files by substituting variable values
fzo - Parse Output files from calculations
fzr - Run complete parametric calculations end-to-end

Installation

Using pip

pip install funz-fz

Using pipx (recommended for CLI tools)

pipx install funz-fz

pipx installs the package in an isolated environment while making the CLI commands (fz, fzi, fzc, fzo, fzr) available globally.

From Source

git clone https://github.com/Funz/fz.git
cd fz
pip install -e .

Or straight from GitHub via pip:

pip install -e git+https://github.com/Funz/fz.git

Dependencies

# Optional dependencies:

# for SSH support
pip install paramiko

# for DataFrame support
pip install pandas

# for R interpreter support
pip install funz-fz[r]
# OR
pip install rpy2
# Note: Requires R installed with system libraries - see examples/r_interpreter_example.md

Quick Start

Here's a complete example for a simple parametric study:

1. Create an Input Template

Create input.txt:

# input file for Perfect Gaz Pressure, with variables n_mol, T_celsius, V_L
n_mol=$n_mol
T_kelvin=@{$T_celsius + 273.15}
#@ def L_to_m3(L):
#@     return(L / 1000)
V_m3=@{L_to_m3($V_L)}

Or using R for formulas (assuming R interpreter is set up: fz.set_interpreter("R")):

# input file for Perfect Gaz Pressure, with variables n_mol, T_celsius, V_L
n_mol=$n_mol
T_kelvin=@{$T_celsius + 273.15}
#@ L_to_m3 <- function(L) {
#@     return (L / 1000)
#@ }
V_m3=@{L_to_m3($V_L)}

2. Create a Calculation Script

Create PerfectGazPressure.sh:

#!/bin/bash

# read input file
source $1

sleep 5 # simulate a calculation time

echo 'pressure = '`echo "scale=4;$n_mol*8.314*$T_kelvin/$V_m3" | bc` > output.txt

echo 'Done'

Make it executable:

chmod +x PerfectGazPressure.sh

3. Run Parametric Study

Create run_study.py:

import fz

# Define the model
model = {
    "varprefix": "$",
    "formulaprefix": "@",
    "delim": "{}",
    "commentline": "#",
    "output": {
        "pressure": "grep 'pressure = ' output.txt | awk '{print $3}'"
    }
}

# Define parameter values
input_variables = {
    "T_celsius": [10, 20, 30, 40],  # 4 temperatures
    "V_L": [1, 2, 5],                # 3 volumes
    "n_mol": 1.0                     # fixed amount
}

# Run all combinations (4 × 3 = 12 cases)
results = fz.fzr(
    "input.txt",
    input_variables,
    model,
    calculators="sh://bash PerfectGazPressure.sh",
    results_dir="results"
)

# Display results
print(results)
print(f"\nCompleted {len(results)} calculations")

Run it:

python run_study.py

Expected output:

   T_celsius  V_L  n_mol     pressure status calculator       error command
0         10  1.0    1.0  235358.1200   done     sh://        None    bash...
1         10  2.0    1.0  117679.0600   done     sh://        None    bash...
2         10  5.0    1.0   47071.6240   done     sh://        None    bash...
3         20  1.0    1.0  243730.2200   done     sh://        None    bash...
...

Completed 12 calculations

CLI Usage

FZ provides command-line tools for quick operations without writing Python scripts. All four core functions are available as CLI commands.

Installation of CLI Tools

The CLI commands are automatically installed when you install the fz package:

pip install -e .

Available commands:

fz - Main entry point (general configuration, plugins management, logging, ...)
fzi - Parse input variables
fzc - Compile input files
fzo - Read output files
fzr - Run parametric calculations
fzl - List and validate installed models and calculators

fzi - Parse Input Variables

Identify variables in input files:

# Parse a single file
fzi input.txt --model perfectgas

# Parse a directory
fzi input_dir/ --model mymodel

# Output formats
fzi input.txt --model perfectgas --format json
fzi input.txt --model perfectgas --format table
fzi input.txt --model perfectgas --format csv

Example:

$ fzi input.txt --model perfectgas --format table
┌──────────────┬───────┐
│ Variable     │ Value │
├──────────────┼───────┤
│ T_celsius    │ None  │
│ V_L          │ None  │
│ n_mol        │ None  │
└──────────────┴───────┘

With inline model definition:

fzi input.txt \
  --varprefix '$' \
  --delim '{}' \
  --format json

Output (JSON):

{
  "T_celsius": null,
  "V_L": null,
  "n_mol": null
}

fzc - Compile Input Files

Substitute variables and create compiled input files:

# Basic usage
fzc input.txt \
  --model perfectgas \
  --variables '{"T_celsius": 25, "V_L": 10, "n_mol": 1}' \
  --output compiled/

# Grid of values (creates subdirectories)
fzc input.txt \
  --model perfectgas \
  --variables '{"T_celsius": [10, 20, 30], "V_L": [1, 2], "n_mol": 1}' \
  --output compiled_grid/

Directory structure created:

compiled_grid/
├── T_celsius=10,V_L=1/
│   └── input.txt
├── T_celsius=10,V_L=2/
│   └── input.txt
├── T_celsius=20,V_L=1/
│   └── input.txt
...

Using formula evaluation:

# Input file with formulas
cat > input.txt << 'EOF'
Temperature: $T_celsius C
#@ T_kelvin = $T_celsius + 273.15
Calculated T: @{T_kelvin} K
EOF

# Compile with formula evaluation
fzc input.txt \
  --varprefix '$' \
  --formulaprefix '@' \
  --delim '{}' \
  --commentline '#' \
  --variables '{"T_celsius": 25}' \
  --output compiled/

fzo - Read Output Files

Parse calculation results:

# Read single directory
fzo results/case1/ --model perfectgas --format table

# Read directory with subdirectories
fzo results/ --model perfectgas --format json

# Different output formats
fzo results/ --model perfectgas --format csv > results.csv
fzo results/ --model perfectgas --format html > results.html
fzo results/ --model perfectgas --format markdown

Example output:

$ fzo results/ --model perfectgas --format table
┌─────────────────────────┬──────────┬────────────┬──────┬───────┐
│ path                    │ pressure │ T_celsius  │ V_L  │ n_mol │
├─────────────────────────┼──────────┼────────────┼──────┼───────┤
│ T_celsius=10,V_L=1      │ 235358.1 │ 10         │ 1.0  │ 1.0   │
│ T_celsius=10,V_L=2      │ 117679.1 │ 10         │ 2.0  │ 1.0   │
│ T_celsius=20,V_L=1      │ 243730.2 │ 20         │ 1.0  │ 1.0   │
└─────────────────────────┴──────────┴────────────┴──────┴───────┘

With inline model definition:

fzo results/ \
  --output-cmd pressure="grep 'pressure = ' output.txt | awk '{print \$3}'" \
  --output-cmd temperature="cat temp.txt" \
  --format json

fzl - List and Validate Models/Calculators

List installed models and calculators with optional validation:

# List all models and calculators
fzl

# List with validation checks
fzl --check

# Filter by pattern
fzl --models "perfect*" --calculators "ssh*"

# Different output formats
fzl --format json
fzl --format table
fzl --format markdown  # default

Example output:

$ fzl --check --format table

=== MODELS ===

Model: perfectgas ✓
  Path: /home/user/project/.fz/models/perfectgas.json
  Supported Calculators: 2
    - local
    - ssh_cluster

Model: navier-stokes ✗
  Path: /home/user/.fz/models/navier-stokes.json
  Error: Missing required field 'output'
  Supported Calculators: 0

=== CALCULATORS ===

Calculator: local ✓
  Path: /home/user/project/.fz/calculators/local.json
  URI: sh://
  Models: 1
    - perfectgas

Calculator: ssh_cluster ✓
  Path: /home/user/.fz/calculators/ssh_cluster.json
  URI: ssh://user@cluster.edu
  Models: 2
    - perfectgas
    - navier-stokes

fzr - Run Parametric Calculations

Execute complete parametric studies from the command line:

# Basic usage
fzr input.txt \
  --model perfectgas \
  --variables '{"T_celsius": [10, 20, 30], "V_L": [1, 2], "n_mol": 1}' \
  --calculator "sh://bash PerfectGazPressure.sh" \
  --results results/

# Multiple calculators for parallel execution
fzr input.txt \
  --model perfectgas \
  --variables '{"param": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}' \
  --calculator "sh://bash calc.sh" \
  --calculator "sh://bash calc.sh" \
  --calculator "sh://bash calc.sh" \
  --results results/ \
  --format table

Using cache:

# First run
fzr input.txt \
  --model perfectgas \
  --variables '{"T_celsius": [10, 20, 30], "V_L": [1, 2]}' \
  --calculator "sh://bash PerfectGazPressure.sh" \
  --results run1/

# Resume with cache (only runs missing cases)
fzr input.txt \
  --model perfectgas \
  --variables '{"T_celsius": [10, 20, 30, 40], "V_L": [1, 2, 3]}' \
  --calculator "cache://run1" \
  --calculator "sh://bash PerfectGazPressure.sh" \
  --results run2/ \
  --format table

Remote SSH execution:

fzr input.txt \
  --model mymodel \
  --variables '{"mesh_size": [100, 200, 400]}' \
  --calculator "ssh://user@cluster.edu/bash /path/to/submit.sh" \
  --results hpc_results/ \
  --format json

Output formats:

# Table (default)
fzr input.txt --model perfectgas --variables '{"x": [1, 2, 3]}' --calculator "sh://calc.sh"

# JSON
fzr ... --format json

# CSV
fzr ... --format csv > results.csv

# Markdown
fzr ... --format markdown

# HTML
fzr ... --format html > results.html

CLI Options Reference

Common Options (all commands)

--help, -h                Show help message
--version                 Show version
--model MODEL             Model alias or inline definition
--varprefix PREFIX        Variable prefix (default: $)
--delim DELIMITERS        Formula delimiters (default: {})
--formulaprefix PREFIX    Formula prefix (default: @)
--commentline CHAR        Comment character (default: #)
--format FORMAT           Output format: json, table, csv, markdown, html

Model Definition Options

Instead of using --model alias, you can define the model inline:

fzr input.txt \
  --varprefix '$' \
  --formulaprefix '@' \
  --delim '{}' \
  --commentline '#' \
  --output-cmd pressure="grep 'pressure' output.txt | awk '{print \$2}'" \
  --output-cmd temp="cat temperature.txt" \
  --variables '{"x": 10}' \
  --calculator "sh://bash calc.sh"

fzr-Specific Options

--calculator URI          Calculator URI (can be specified multiple times)
--results DIR             Results directory (default: results)

Complete CLI Examples

Example 1: Quick Variable Discovery

# Check what variables are in your input files
$ fzi simulation_template.txt --varprefix '$' --format table
┌──────────────┬───────┐
│ Variable     │ Value │
├──────────────┼───────┤
│ mesh_size    │ None  │
│ timestep     │ None  │
│ iterations   │ None  │
└──────────────┴───────┘

Example 2: Quick Compilation Test

# Test variable substitution
$ fzc simulation_template.txt \
  --varprefix '$' \
  --variables '{"mesh_size": 100, "timestep": 0.01, "iterations": 1000}' \
  --output test_compiled/

$ cat test_compiled/simulation_template.txt
# Compiled with mesh_size=100
mesh_size=100
timestep=0.01
iterations=1000

Example 3: Parse Existing Results

# Extract results from previous calculations
$ fzo old_results/ \
  --output-cmd energy="grep 'Total Energy' log.txt | awk '{print \$3}'" \
  --output-cmd time="grep 'CPU Time' log.txt | awk '{print \$3}'" \
  --format csv > analysis.csv

Example 4: End-to-End Parametric Study

#!/bin/bash
# run_study.sh - Complete parametric study from CLI

# 1. Parse input to verify variables
echo "Step 1: Parsing input variables..."
fzi input.txt --model perfectgas --format table

# 2. Run parametric study
echo -e "\nStep 2: Running calculations..."
fzr input.txt \
  --model perfectgas \
  --variables '{
    "T_celsius": [10, 20, 30, 40, 50],
    "V_L": [1, 2, 5, 10],
    "n_mol": 1
  }' \
  --calculator "sh://bash PerfectGazPressure.sh" \
  --calculator "sh://bash PerfectGazPressure.sh" \
  --results results/ \
  --format table

# 3. Export results to CSV
echo -e "\nStep 3: Exporting results..."
fzo results/ --model perfectgas --format csv > results.csv
echo "Results saved to results.csv"

Example 5: Using Model and Calculator Aliases

First, create model and calculator configurations:

# Create model alias
mkdir -p .fz/models
cat > .fz/models/perfectgas.json << 'EOF'
{
  "varprefix": "$",
  "formulaprefix": "@",
  "delim": "{}",
  "commentline": "#",
  "output": {
    "pressure": "grep 'pressure = ' output.txt | awk '{print $3}'"
  },
  "id": "perfectgas"
}
EOF

# Create calculator alias
mkdir -p .fz/calculators
cat > .fz/calculators/local.json << 'EOF'
{
  "uri": "sh://",
  "models": {
    "perfectgas": "bash PerfectGazPressure.sh"
  }
}
EOF

# Now run with short aliases
fzr input.txt \
  --model perfectgas \
  --variables '{"T_celsius": [10, 20, 30], "V_L": [1, 2]}' \
  --calculator local \
  --results results/ \
  --format table

Example 6: Interrupt and Resume

# Start long-running calculation
fzr input.txt \
  --model mymodel \
  --variables '{"param": [1..100]}' \
  --calculator "sh://bash slow_calc.sh" \
  --results run1/
# Press Ctrl+C after some cases complete...
# ⚠️  Interrupt received (Ctrl+C). Gracefully shutting down...
# ⚠️  Execution was interrupted. Partial results may be available.

# Resume from cache
fzr input.txt \
  --model mymodel \
  --variables '{"param": [1..100]}' \
  --calculator "cache://run1" \
  --calculator "sh://bash slow_calc.sh" \
  --results run1_resumed/ \
  --format table
# Only runs the remaining cases

Environment Variables for CLI

# Set logging level
export FZ_LOG_LEVEL=DEBUG
fzr input.txt --model perfectgas ...

# Set maximum parallel workers
export FZ_MAX_WORKERS=4
fzr input.txt --model perfectgas --calculator "sh://calc.sh" ...

# Set retry attempts
export FZ_MAX_RETRIES=3
fzr input.txt --model perfectgas ...

# SSH configuration
export FZ_SSH_AUTO_ACCEPT_HOSTKEYS=1  # Use with caution
export FZ_SSH_KEEPALIVE=300
fzr input.txt --calculator "ssh://user@host/bash calc.sh" ...

Core Functions

fzi - Parse Input Variables

Identify all variables in an input file or directory:

import fz

model = {
    "varprefix": "$",
    "delim": "{}"
}

# Parse single file
variables = fz.fzi("input.txt", model)
# Returns: {'T_celsius': None, 'V_L': None, 'n_mol': None}

# Parse directory (scans all files)
variables = fz.fzi("input_dir/", model)

Returns: Dictionary with variable names as keys (values are None)

fzc - Compile Input Files

Substitute variable values and evaluate formulas:

import fz

model = {
    "varprefix": "$",
    "formulaprefix": "@",
    "delim": "{}",
    "commentline": "#"
}

input_variables = {
    "T_celsius": 25,
    "V_L": 10,
    "n_mol": 2
}

# Compile single file
fz.fzc(
    "input.txt",
    input_variables,
    model,
    output_dir="compiled"
)

# Compile with multiple value sets (creates subdirectories)
fz.fzc(
    "input.txt",
    {
        "T_celsius": [20, 30],  # 2 values
        "V_L": [5, 10],         # 2 values
        "n_mol": 1              # fixed
    },
    model,
    output_dir="compiled_grid"
)
# Creates: compiled_grid/T_celsius=20,V_L=5/, T_celsius=20,V_L=10/, etc.

Parameters:

input_path: Path to input file or directory
input_variables: Dictionary of variable values (scalar or list)
model: Model definition (dict or alias name)
output_dir: Output directory path

fzo - Read Output Files

Parse calculation results from output directory:

import fz

model = {
    "output": {
        "pressure": "grep 'Pressure:' output.txt | awk '{print $2}'",
        "temperature": "grep 'Temperature:' output.txt | awk '{print $2}'"
    }
}

# Read from single directory
output = fz.fzo("results/case1", model)
# Returns: DataFrame with 1 row

# Read from directory with subdirectories
output = fz.fzo("results/*", model)
# Returns: DataFrame with 1 row per subdirectory

Automatic Path Parsing: If subdirectory names follow the pattern key1=val1,key2=val2,..., variables are automatically extracted as columns:

# Directory structure:
# results/
#   ├── T_celsius=20,V_L=1/output.txt
#   ├── T_celsius=20,V_L=2/output.txt
#   └── T_celsius=30,V_L=1/output.txt

output = fz.fzo("results/*", model)
print(output)
#                          path  pressure  T_celsius  V_L
# 0  T_celsius=20,V_L=1  2437.30       20.0  1.0
# 1  T_celsius=20,V_L=2  1218.65       20.0  2.0
# 2  T_celsius=30,V_L=1  2520.74       30.0  1.0

fzr - Run Parametric Calculations

Execute complete parametric study with automatic parallelization:

import fz

model = {
    "varprefix": "$",
    "output": {
        "result": "cat output.txt"
    }
}

results = fz.fzr(
    input_path="input.txt",
    input_variables={
        "temperature": [100, 200, 300],
        "pressure": [1, 10, 100],
        "concentration": 0.5
    },
    model=model,
    calculators=["sh://bash calculate.sh"],
    results_dir="results"
)

# Results DataFrame includes:
# - All variable columns
# - All output columns
# - Metadata: status, calculator, error, command
print(results)

Parameters:

input_path: Input file or directory path
input_variables: Variable values (creates Cartesian product of lists)
model: Model definition (dict or alias)
calculators: Calculator URI(s) - string or list
results_dir: Results directory path

Returns: pandas DataFrame with all results

Model Definition

A model defines how to parse inputs and extract outputs:

model = {
    # Input parsing
    "varprefix": "$",           # Variable marker (e.g., $temp)
    "formulaprefix": "@",       # Formula marker (e.g., @pressure)
    "delim": "{}",              # Formula delimiters
    "commentline": "#",         # Comment character

    # Optional: formula interpreter
    "interpreter": "python",    # "python" (default) or "R"

    # Output extraction (shell commands)
    "output": {
        "pressure": "grep 'P =' out.txt | awk '{print $3}'",
        "temperature": "cat temp.txt",
        "energy": "python extract.py"
    },

    # Optional: model identifier
    "id": "perfectgas"
}

Model Aliases

Store reusable models in .fz/models/:

.fz/models/perfectgas.json:

{
    "varprefix": "$",
    "formulaprefix": "@",
    "delim": "{}",
    "commentline": "#",
    "output": {
        "pressure": "grep 'pressure = ' output.txt | awk '{print $3}'"
    },
    "id": "perfectgas"
}

Use by name:

results = fz.fzr("input.txt", input_variables, "perfectgas")

Formula Evaluation

Formulas in input files are evaluated during compilation using Python or R interpreters.

Python Interpreter (Default)

# Input template with formulas
Temperature: $T_celsius C
Volume: $V_L L

# Context (available in all formulas)
#@import math
#@R = 8.314
#@def celsius_to_kelvin(t):
#@    return t + 273.15

# Calculated value
#@T_kelvin = celsius_to_kelvin($T_celsius)
#@pressure = $n_mol * R * T_kelvin / ($V_L / 1000)

Result: @{pressure} Pa
Circumference: @{2 * math.pi * $radius}

R Interpreter

For statistical computing, you can use R for formula evaluation:

from fz import fzi
from fz.config import set_interpreter

# Set interpreter to R
set_interpreter("R")

# Or specify in model
model = {"interpreter": "R", "formulaprefix": "@", "delim": "{}", "commentline": "#"}

R template example:

# Input template with R formulas
Sample size: $n
Mean: $mu
SD: $sigma

# R context (available in all formulas)
#@samples <- rnorm($n, mean=$mu, sd=$sigma)

Mean (sample): @{mean(samples)}
SD (sample): @{sd(samples)}
Median: @{median(samples)}

Installation requirements: R must be installed along with system libraries. See examples/r_interpreter_example.md for detailed installation instructions.

# Install with R support
pip install funz-fz[r]

Key differences:

Python requires import math for math.pi, R has pi built-in
R excels at statistical functions: mean(), sd(), median(), rnorm(), etc.
R uses <- for assignment in context lines
R is vectorized by default

Variable Default Values

Variables can specify default values using the ${var~default} syntax:

# Configuration template
Host: ${host~localhost}
Port: ${port~8080}
Debug: ${debug~false}
Workers: ${workers~4}

Behavior:

If variable is provided in input_variables, its value is used
If variable is NOT provided but has default, default is used (with warning)
If variable is NOT provided and has NO default, it remains unchanged

Example:

from fz.interpreter import replace_variables_in_content

content = "Server: ${host~localhost}:${port~8080}"
input_variables = {"host": "example.com"}  # port not provided

result = replace_variables_in_content(content, input_variables)
# Result: "Server: example.com:8080"
# Warning: Variable 'port' not found in input_variables, using default value: '8080'

Use cases:

Configuration templates with sensible defaults
Environment-specific deployments
Optional parameters in parametric studies

See examples/variable_substitution.md for comprehensive documentation.

Features:

Python or R expression evaluation
Multi-line function definitions
Variable substitution in formulas
Default values for variables
Nested formula evaluation

Calculator Types

Local Shell Execution

Execute calculations locally:

# Basic shell command
calculators = "sh://bash script.sh"

# With multiple arguments
calculators = "sh://python calculate.py --verbose"

# Multiple calculators (tries in order, parallel execution)
calculators = [
    "sh://bash method1.sh",
    "sh://bash method2.sh",
    "sh://python method3.py"
]

How it works:

Input files copied to temporary directory
Command executed in that directory with input files as arguments
Outputs parsed from result directory
Temporary files cleaned up (preserved in DEBUG mode)

SSH Remote Execution

Execute calculations on remote servers:

# SSH with password
calculators = "ssh://user:password@server.com:22/bash /absolutepath/to/calc.sh"

# SSH with key-based auth (recommended)
calculators = "ssh://user@server.com/bash /absolutepath/to/calc.sh"

# SSH with custom port
calculators = "ssh://user@server.com:2222/bash /absolutepath/to/calc.sh"

Features:

Automatic file transfer (SFTP)
Remote execution with timeout
Result retrieval
SSH key-based or password authentication
Host key verification

Security:

Interactive host key acceptance
Warning for password-based auth
Environment variable for auto-accepting host keys: FZ_SSH_AUTO_ACCEPT_HOSTKEYS=1

SLURM Workload Manager

Execute calculations on SLURM clusters (local or remote):

# Local SLURM execution
calculators = "slurm://:compute/bash script.sh"

# Remote SLURM execution via SSH
calculators = "slurm://user@cluster.edu:gpu/bash script.sh"

# With custom SSH port
calculators = "slurm://user@cluster.edu:2222:gpu/bash script.sh"

# Multiple partitions for parallel execution
calculators = [
    "slurm://user@hpc.edu:compute/bash calc.sh",
    "slurm://user@hpc.edu:gpu/bash calc.sh"
]

URI Format: slurm://[user@host[:port]]:partition/script

Note: For local execution, the partition must be prefixed with a colon (:partition), e.g., slurm://:compute/script.sh

How it works:

Local execution: Uses srun --partition=<partition> <script> directly
Remote execution: Connects via SSH, transfers files, runs srun on remote cluster
Automatically handles SLURM partition scheduling
Supports interrupt handling (Ctrl+C terminates SLURM jobs)

Features:

Local or remote SLURM execution
Automatic file transfer for remote execution (via SFTP)
SLURM partition specification
Timeout and interrupt handling
Compatible with all SLURM schedulers

Requirements:

Local: SLURM installed (srun command available)
Remote: SSH access to SLURM cluster + paramiko library

Funz Server Execution

Execute calculations using the Funz server protocol (compatible with legacy Java Funz servers):

# Connect to local Funz server
calculators = "funz://:5555/R"

# Connect to remote Funz server
calculators = "funz://server.example.com:5555/Python"

# Multiple Funz servers for parallel execution
calculators = [
    "funz://:5555/R",
    "funz://:5556/R",
    "funz://:5557/R"
]

Features:

Compatible with legacy Java Funz calculator servers
Automatic file upload to server
Remote execution with the Funz protocol
Result download and extraction
Support for interrupt handling

Protocol:

Text-based TCP socket communication
Calculator reservation with authentication
Automatic cleanup and unreservation

URI Format: funz://[host]:<port>/<code>

host: Server hostname (default: localhost)
port: Server port (required)
code: Calculator code/model name (e.g., "R", "Python", "Modelica")

Example:

import fz

model = {
    "output": {
        "pressure": "grep 'pressure = ' output.txt | awk '{print $3}'"
    }
}

results = fz.fzr(
    "input.txt",
    {"temp": [100, 200, 300]},
    model,
    calculators="funz://:5555/R"
)

Cache Calculator

Reuse previous calculation results:

# Check single cache directory
calculators = "cache://previous_results"

# Check multiple cache locations
calculators = [
    "cache://run1",
    "cache://run2/results",
    "sh://bash calculate.sh"  # Fallback to actual calculation
]

# Use glob patterns
calculators = "cache://archive/*/results"

Cache Matching:

Based on MD5 hash of input files (.fz_hash)
Validates outputs are not None
Falls through to next calculator on miss
No recalculation if cache hit

Calculator Aliases

Store calculator configurations in .fz/calculators/:

.fz/calculators/cluster.json:

{
    "uri": "ssh://user@cluster.university.edu",
    "models": {
        "perfectgas": "bash /home/user/codes/perfectgas/run.sh",
        "navier-stokes": "bash /home/user/codes/cfd/run.sh"
    }
}

Use by name:

results = fz.fzr("input.txt", input_variables, "perfectgas", calculators="cluster")

Advanced Features

Parallel Execution

FZ automatically parallelizes when you have multiple cases and calculators:

# Sequential: 1 calculator, 10 cases → runs one at a time
results = fz.fzr(
    "input.txt",
    {"temp": list(range(10))},
    model,
    calculators="sh://bash calc.sh"
)

# Parallel: 3 calculators, 10 cases → 3 concurrent
results = fz.fzr(
    "input.txt",
    {"temp": list(range(10))},
    model,
    calculators=[
        "sh://bash calc.sh",
        "sh://bash calc.sh",
        "sh://bash calc.sh"
    ]
)

# Control parallelism with environment variable
import os
os.environ['FZ_MAX_WORKERS'] = '4'

# Or use duplicate calculator URIs
calculators = ["sh://bash calc.sh"] * 4  # 4 parallel workers

Load Balancing:

Round-robin distribution of cases to calculators
Thread-safe calculator locking
Automatic retry on failures
Progress tracking with ETA

Retry Mechanism

Automatic retry on calculation failures:

import os
os.environ['FZ_MAX_RETRIES'] = '3'  # Try each case up to 3 times

results = fz.fzr(
    "input.txt",
    input_variables,
    model,
    calculators=[
        "sh://unreliable_calc.sh",  # Might fail
        "sh://backup_calc.sh"        # Backup method
    ]
)

Retry Strategy:

Try first available calculator
On failure, try next calculator
Repeat up to FZ_MAX_RETRIES times
Report all attempts in logs

Caching Strategy

Intelligent result reuse:

# First run
results1 = fz.fzr(
    "input.txt",
    {"temp": [10, 20, 30]},
    model,
    calculators="sh://expensive_calc.sh",
    results_dir="run1"
)

# Add more cases - reuse previous results
results2 = fz.fzr(
    "input.txt",
    {"temp": [10, 20, 30, 40, 50]},  # 2 new cases
    model,
    calculators=[
        "cache://run1",              # Check cache first
        "sh://expensive_calc.sh"     # Only run new cases
    ],
    results_dir="run2"
)
# Only runs calculations for temp=40 and temp=50

Output Type Casting

Automatic type conversion:

model = {
    "output": {
        "scalar_int": "echo 42",
        "scalar_float": "echo 3.14159",
        "array": "echo '[1, 2, 3, 4, 5]'",
        "single_array": "echo '[42]'",  # → 42 (simplified)
        "json_object": "echo '{\"key\": \"value\"}'",
        "string": "echo 'hello world'"
    }
}

results = fz.fzo("output_dir", model)
# Values automatically cast to int, float, list, dict, or str

Casting Rules:

Try JSON parsing
Try Python literal evaluation
Try numeric conversion (int/float)
Keep as string
Single-element arrays → scalar

Complete Examples

Example 1: Perfect Gas Pressure Study

Input file (input.txt):

# input file for Perfect Gaz Pressure, with variables n_mol, T_celsius, V_L
n_mol=$n_mol
T_kelvin=@{$T_celsius + 273.15}
#@ def L_to_m3(L):
#@     return(L / 1000)
V_m3=@{L_to_m3($V_L)}

Calculation script (PerfectGazPressure.sh):

#!/bin/bash

# read input file
source $1

sleep 5 # simulate a calculation time

echo 'pressure = '`echo "scale=4;$n_mol*8.314*$T_kelvin/$V_m3" | bc` > output.txt

echo 'Done'

Python script (run_perfectgas.py):

import fz
import matplotlib.pyplot as plt

# Define model
model = {
    "varprefix": "$",
    "formulaprefix": "@",
    "delim": "{}",
    "commentline": "#",
    "output": {
        "pressure": "grep 'pressure = ' output.txt | awk '{print $3}'"
    }
}

# Parametric study
results = fz.fzr(
    "input.txt",
    {
        "n_mol": [1, 2, 3],
        "T_celsius": [10, 20, 30],
        "V_L": [5, 10]
    },
    model,
    calculators="sh://bash PerfectGazPressure.sh",
    results_dir="perfectgas_results"
)

print(results)

# Plot results: pressure vs temperature for different volumes
for volume in results['V_L'].unique():
    for n in results['n_mol'].unique():
        data = results[(results['V_L'] == volume) & (results['n_mol'] == n)]
        plt.plot(data['T_celsius'], data['pressure'],
                 marker='o', label=f'n={n} mol, V={volume} L')

plt.xlabel('Temperature (°C)')
plt.ylabel('Pressure (Pa)')
plt.title('Ideal Gas: Pressure vs Temperature')
plt.legend()
plt.grid(True)
plt.savefig('perfectgas_results.png')
print("Plot saved to perfectgas_results.png")

Example 2: Remote HPC Calculation

import fz

model = {
    "varprefix": "$",
    "output": {
        "energy": "grep 'Total Energy' output.log | awk '{print $4}'",
        "time": "grep 'CPU time' output.log | awk '{print $4}'"
    }
}

# Run on HPC cluster
results = fz.fzr(
    "simulation_input/",
    {
        "mesh_size": [100, 200, 400, 800],
        "timestep": [0.001, 0.01, 0.1],
        "iterations": 1000
    },
    model,
    calculators=[
        "cache://previous_runs/*",  # Check cache first
        "ssh://user@hpc.university.edu/sbatch /path/to/submit.sh"
    ],
    results_dir="hpc_results"
)

# Analyze convergence
import pandas as pd
summary = results.groupby('mesh_size').agg({
    'energy': ['mean', 'std'],
    'time': 'sum'
})
print(summary)

Example 3: Multi-Calculator with Failover

import fz

model = {
    "varprefix": "$",
    "output": {"result": "cat result.txt"}
}

results = fz.fzr(
    "input.txt",
    {"param": list(range(100))},
    model,
    calculators=[
        "cache://previous_results",           # 1. Check cache
        "sh://bash fast_but_unstable.sh",    # 2. Try fast method
        "sh://bash robust_method.sh",        # 3. Fallback to robust
        "ssh://user@server/bash remote.sh"   # 4. Last resort: remote
    ],
    results_dir="results"
)

# Check which calculator was used for each case
print(results[['param', 'calculator', 'status']].head(10))

Configuration

Environment Variables

# Logging level (DEBUG, INFO, WARNING, ERROR)
export FZ_LOG_LEVEL=INFO

# Maximum retry attempts per case
export FZ_MAX_RETRIES=5

# Thread pool size for parallel execution
export FZ_MAX_WORKERS=8

# SSH keepalive interval (seconds)
export FZ_SSH_KEEPALIVE=300

# Auto-accept SSH host keys (use with caution!)
export FZ_SSH_AUTO_ACCEPT_HOSTKEYS=0

# Default formula interpreter (python or R)
export FZ_INTERPRETER=python

# Custom shell binary search path (overrides system PATH)
# Windows example: SET FZ_SHELL_PATH=C:\msys64\usr\bin;C:\Program Files\Git\usr\bin
# Linux/macOS example: export FZ_SHELL_PATH=/opt/custom/bin:/usr/local/bin
export FZ_SHELL_PATH=/usr/local/bin:/usr/bin

# Execution timeout in seconds (default per calculator)
export FZ_EXECUTION_TIMEOUT=3600

Shell Path Configuration (FZ_SHELL_PATH)

The FZ_SHELL_PATH environment variable allows you to specify custom locations for shell binaries (grep, awk, sed, etc.) used in model output expressions and calculator commands. This is particularly important on Windows where Unix-like tools may be installed in non-standard locations.

Why use FZ_SHELL_PATH?

Windows compatibility: Locate tools in MSYS2, Git Bash, Cygwin, or WSL
Custom installations: Use specific versions of tools from custom directories
Priority control: Override system PATH to ensure correct tool versions
Performance: Cached binary paths for faster resolution

Usage examples:

# Windows with MSYS2 (use semicolon separator)
SET FZ_SHELL_PATH=C:\msys64\usr\bin;C:\msys64\mingw64\bin

# Windows with Git Bash
SET FZ_SHELL_PATH=C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin

# Linux/macOS (use colon separator)
export FZ_SHELL_PATH=/opt/homebrew/bin:/usr/local/bin

# Priority: FZ_SHELL_PATH paths are checked BEFORE system PATH

How it works:

Commands in model output dictionaries are parsed for binary names (grep, awk, etc.)
Binary names are resolved to absolute paths using FZ_SHELL_PATH
Commands in sh:// calculators are similarly resolved
Windows: Automatically tries both command and command.exe
Resolved paths are cached for performance

Example in model:

model = {
    "output": {
        "pressure": "grep 'pressure' output.txt | awk '{print $2}'"
    }
}
# With FZ_SHELL_PATH=C:\msys64\usr\bin, executes:
# C:\msys64\usr\bin\grep.exe 'pressure' output.txt | C:\msys64\usr\bin\awk.exe '{print $2}'

See SHELL_PATH_IMPLEMENTATION.md and examples/shell_path_example.md for detailed documentation.

Python Configuration

from fz import get_config

# Get current config
config = get_config()
print(f"Max retries: {config.max_retries}")
print(f"Max workers: {config.max_workers}")

# Modify configuration
config.max_retries = 10
config.max_workers = 4

Directory Structure

FZ uses the following directory structure:

your_project/
├── input.txt                 # Your input template
├── calculate.sh              # Your calculation script
├── run_study.py             # Your Python script
├── .fz/                     # FZ configuration (optional)
│   ├── models/              # Model aliases
│   │   └── mymodel.json
│   ├── calculators/         # Calculator aliases
│   │   └── mycluster.json
│   └── tmp/                 # Temporary files (auto-created)
│       └── fz_temp_*/       # Per-run temp directories
└── results/                 # Results directory
    ├── case1/               # One directory per case
    │   ├── input.txt        # Compiled input
    │   ├── output.txt       # Calculation output
    │   ├── log.txt          # Execution metadata
    │   ├── out.txt          # Standard output
    │   ├── err.txt          # Standard error
    │   └── .fz_hash         # File checksums (for caching)
    └── case2/
        └── ...

Interrupt Handling

FZ supports graceful interrupt handling for long-running calculations:

How to Interrupt

Press Ctrl+C during execution:

python run_study.py
# ... calculations running ...
# Press Ctrl+C
⚠️  Interrupt received (Ctrl+C). Gracefully shutting down...
⚠️  Press Ctrl+C again to force quit (not recommended)

What Happens

First Ctrl+C:
- Currently running calculations complete
- No new calculations start
- Partial results are saved
- Resources are cleaned up
- Signal handlers restored
Second Ctrl+C (not recommended):
- Immediate termination
- May leave resources in inconsistent state

Resuming After Interrupt

Use caching to resume from where you left off:

# First run (interrupted after 50/100 cases)
results1 = fz.fzr(
    "input.txt",
    {"param": list(range(100))},
    model,
    calculators="sh://bash calc.sh",
    results_dir="results"
)
print(f"Completed {len(results1)} cases before interrupt")

# Resume using cache
results2 = fz.fzr(
    "input.txt",
    {"param": list(range(100))},
    model,
    calculators=[
        "cache://results",      # Reuse completed cases
        "sh://bash calc.sh"     # Run remaining cases
    ],
    results_dir="results_resumed"
)
print(f"Total completed: {len(results2)} cases")

Example with Interrupt Handling

import fz
import signal
import sys

model = {
    "varprefix": "$",
    "output": {"result": "cat output.txt"}
}

def main():
    try:
        results = fz.fzr(
            "input.txt",
            {"param": list(range(1000))},  # Many cases
            model,
            calculators="sh://bash slow_calculation.sh",
            results_dir="results"
        )

        print(f"\n✅ Completed {len(results)} calculations")
        return results

    except KeyboardInterrupt:
        # This should rarely happen (graceful shutdown handles it)
        print("\n❌ Forcefully terminated")
        sys.exit(1)

if __name__ == "__main__":
    main()

Output File Structure

Each case creates a directory with complete execution metadata:

`log.txt` - Execution Metadata

Command: bash calculate.sh input.txt
Exit code: 0
Time start: 2024-03-15T10:30:45.123456
Time end: 2024-03-15T10:32:12.654321
Execution time: 87.531 seconds
User: john_doe
Hostname: compute-01
Operating system: Linux
Platform: Linux-5.15.0-x86_64
Working directory: /tmp/fz_temp_abc123/case1
Original directory: /home/john/project

`.fz_hash` - Input File Checksums

a1b2c3d4e5f6...  input.txt
f6e5d4c3b2a1...  config.dat

Used for cache matching.

Development

Running Tests

# Install development dependencies
pip install -e .[dev]

# Run all tests
python -m pytest tests/ -v

# Run specific test file
python -m pytest tests/test_examples_perfectgaz.py -v

# Run with debug output
FZ_LOG_LEVEL=DEBUG python -m pytest tests/test_parallel.py -v

# Run tests matching pattern
python -m pytest tests/ -k "parallel" -v

# Test interrupt handling
python -m pytest tests/test_interrupt_handling.py -v

# Run examples
python example_usage.py
python example_interrupt.py  # Interactive interrupt demo

Project Structure

fz/
├── fz/                          # Main package
│   ├── __init__.py              # Public API exports
│   ├── core.py                  # Core functions (fzi, fzc, fzo, fzr)
│   ├── interpreter.py                # Variable parsing, formula evaluation
│   ├── runners.py               # Calculation execution (sh, ssh)
│   ├── helpers.py               # Parallel execution, retry logic
│   ├── io.py                    # File I/O, caching, hashing
│   ├── logging.py               # Logging configuration
│   └── config.py                # Configuration management
├── tests/                       # Test suite
│   ├── test_parallel.py         # Parallel execution tests
│   ├── test_interrupt_handling.py  # Interrupt handling tests
│   ├── test_examples_*.py       # Example-based tests
│   └── ...
├── README.md                    # This file
└── setup.py                     # Package configuration

Testing Your Own Models

Create a test following this pattern:

import fz
import tempfile
from pathlib import Path

def test_my_model():
    # Create input
    with tempfile.TemporaryDirectory() as tmpdir:
        input_file = Path(tmpdir) / "input.txt"
        input_file.write_text("Parameter: $param\n")

        # Create calculator script
        calc_script = Path(tmpdir) / "calc.sh"
        calc_script.write_text("""#!/bin/bash
source $1
echo "result=$param" > output.txt
""")
        calc_script.chmod(0o755)

        # Define model
        model = {
            "varprefix": "$",
            "output": {
                "result": "grep 'result=' output.txt | cut -d= -f2"
            }
        }

        # Run test
        results = fz.fzr(
            str(input_file),
            {"param": [1, 2, 3]},
            model,
            calculators=f"sh://bash {calc_script}",
            results_dir=str(Path(tmpdir) / "results")
        )

        # Verify
        assert len(results) == 3
        assert list(results['result']) == [1, 2, 3]
        assert all(results['status'] == 'done')

        print("✅ Test passed!")

if __name__ == "__main__":
    test_my_model()

Troubleshooting

Common Issues

Problem: Calculations fail with "command not found"

# Solution: Use absolute paths in calculator URIs
calculators = "sh://bash /full/path/to/script.sh"

Problem: SSH calculations hang

# Solution: Increase timeout or check SSH connectivity
calculators = "ssh://user@host/bash script.sh"
# Test manually: ssh user@host "bash script.sh"

Problem: Cache not working

# Solution: Check .fz_hash files exist in cache directories
# Enable debug logging to see cache matching process
import os
os.environ['FZ_LOG_LEVEL'] = 'DEBUG'

Problem: Out of memory with many parallel cases

# Solution: Limit parallel workers
export FZ_MAX_WORKERS=2

Debug Mode

Enable detailed logging:

import os
os.environ['FZ_LOG_LEVEL'] = 'DEBUG'

results = fz.fzr(...)  # Will show detailed execution logs

Debug output includes:

Calculator selection and locking
File operations
Command execution
Cache matching
Thread pool management
Temporary directory preservation

Performance Tips

Use caching: Reuse previous results when possible
Limit parallelism: Don't exceed your CPU/memory limits
Optimize calculators: Fast calculators first in the list
Batch similar cases: Group cases that use the same calculator
Use SSH keepalive: For long-running remote calculations
Clean old results: Remove old result directories to save disk space

License

BSD 3-Clause License. See LICENSE file for details.

Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch
Add tests for new features
Ensure all tests pass
Submit a pull request

Citation

If you use FZ in your research, please cite:

@software{fz,
  title = {FZ: Parametric Scientific Computing Framework},
  designers = {[Yann Richet]},
  authors = {[Claude Sonnet, Yann Richet]},
  year = {2025},
  url = {https://github.com/Funz/fz}
}

Support

Issues: https://github.com/Funz/fz/issues
Documentation: https://fz.github.io
Examples: See tests/test_examples_*.py for working examples

Name		Name	Last commit message	Last commit date
Latest commit History 233 Commits
.github		.github
completions		completions
context		context
examples		examples
fz		fz
scripts		scripts
tests		tests
tools		tools
.gitignore		.gitignore
LICENSE		LICENSE
NEWS.md		NEWS.md
README.md		README.md
pyproject.toml		pyproject.toml
setup.py		setup.py

License

Funz/fz

Folders and files

Latest commit

History

Repository files navigation

FZ - Parametric Scientific Computing Framework

Table of Contents

Features

Core Capabilities

Four Core Functions

Installation

Using pip

Using pipx (recommended for CLI tools)

From Source

Dependencies

Quick Start

1. Create an Input Template

2. Create a Calculation Script

3. Run Parametric Study

CLI Usage

Installation of CLI Tools

fzi - Parse Input Variables

fzc - Compile Input Files

fzo - Read Output Files

fzl - List and Validate Models/Calculators

fzr - Run Parametric Calculations

CLI Options Reference

Common Options (all commands)

Model Definition Options

fzr-Specific Options

Complete CLI Examples

Example 1: Quick Variable Discovery

Example 2: Quick Compilation Test

Example 3: Parse Existing Results

Example 4: End-to-End Parametric Study

Example 5: Using Model and Calculator Aliases

Example 6: Interrupt and Resume

Environment Variables for CLI

Core Functions

fzi - Parse Input Variables

fzc - Compile Input Files

fzo - Read Output Files

fzr - Run Parametric Calculations

Model Definition

Model Aliases

Formula Evaluation

Python Interpreter (Default)

R Interpreter

Variable Default Values

Calculator Types

Local Shell Execution

SSH Remote Execution

SLURM Workload Manager

Funz Server Execution

Cache Calculator

Calculator Aliases

Advanced Features

Parallel Execution

Retry Mechanism

Caching Strategy

Output Type Casting

Complete Examples

Example 1: Perfect Gas Pressure Study

Example 2: Remote HPC Calculation

Example 3: Multi-Calculator with Failover

Configuration

Environment Variables

Shell Path Configuration (FZ_SHELL_PATH)

Python Configuration

Directory Structure

Interrupt Handling

How to Interrupt

What Happens

Resuming After Interrupt

Example with Interrupt Handling

Output File Structure

log.txt - Execution Metadata

.fz_hash - Input File Checksums

Development

`log.txt` - Execution Metadata

`.fz_hash` - Input File Checksums

Packages