SyntheticData - Enhanced OHLCV Synthetic Data Generator

A powerful Python library for generating synthetic financial time series data using SDV's PARSynthesizer. Enhanced with AI-agent friendly APIs, structured configuration, and modern Python practices.

🎯 Features

• Generate Synthetic OHLCV Data: Create realistic stock market data using advanced sequential synthesis
• AI Agent Integration: Clean, structured APIs designed for AI agents and automated systems
• Flexible Model Storage: Organized model persistence in ticker-specific folders
• Batch Processing: Train multiple models and generate scenarios efficiently
• Backward Compatible: Full compatibility with existing codebases
• Comprehensive Testing: Validated with extensive test suite

🚀 Quick Start

Installation

1. Clone the Repository:

git clone https://github.com/khanhnl2806/syntheticdata.git
cd syntheticdata

2. Set Up Virtual Environment:

# Create virtual environment
python -m venv .venv-synthetic

# Activate virtual environment
.venv-synthetic\Scripts\activate

3. Install Dependencies:

pip install -r requirements.txt
# or if using uv:
uv pip install -r requirements.txt

4. Install Package in Development Mode:

pip install -e .

Quick Example

from syntheticdata import SyntheticDataAPI
import pandas as pd

# Initialize API
api = SyntheticDataAPI()

# Train a model (requires OHLCV1D.parquet in data_folder/SYMBOL/)
result = api.train_single_model(
    symbol="ACB",
    data_folder="./data",
    start_date="2020-01-01",
    end_date="2024-01-01"
)

# Generate synthetic scenarios
scenarios = api.generate_single_scenario(
    symbol="ACB",
    data_folder="./data", 
    seeds=[1, 2, 3],
    steps=252,  # 1 year of daily data
    save_results=True
)

📚 Documentation

Project Structure

syntheticdata/
├── src/syntheticdata/           # Main package
│   ├── __init__.py             # Package interface
│   ├── agent_api.py            # AI-friendly API
│   ├── config.py               # Configuration classes
│   ├── generator.py            # Core generator (original)
│   ├── main.py                 # CLI interface
│   ├── services.py             # Business logic layer
│   └── utils.py                # Utility functions
├── artifacts/                   # Pre-trained models (97 Vietnamese stocks)
├── examples/                    # Usage examples
├── tests/                      # Test suite
├── notebook/                   # Jupyter notebooks
├── requirements.txt            # Dependencies
├── pyproject.toml             # Project configuration
└── README.md                  # This file

Data Format

Your data should be organized as:

data_folder/
├── SYMBOL1/
│   └── OHLCV1D.parquet        # Daily OHLCV data
├── SYMBOL2/
│   └── OHLCV1D.parquet
└── ...

Required columns in OHLCV1D.parquet:

• open, high, low, close, volume
• DatetimeIndex for dates

API Overview

SyntheticDataAPI (Recommended for AI Agents)

from syntheticdata import SyntheticDataAPI

api = SyntheticDataAPI()

# Core methods:
api.train_single_model(symbol, data_folder, start_date, end_date)
api.generate_single_scenario(symbol, data_folder, seeds, steps)
api.batch_generate(symbols, data_folder, seeds, steps)
api.check_model_status(symbol, data_folder, end_date)
api.get_available_symbols(data_folder)

Quick Functions (For Simple Use Cases)

from syntheticdata import quick_train, quick_generate

# Train a model quickly
result = quick_train("ACB", "./data")

# Generate scenarios quickly  
scenarios = quick_generate("ACB", "./data", seeds=[1, 2, 3])

Original Generator (For Advanced Users)

from syntheticdata import SyntheticOHLCVGenerator
import pandas as pd

# Load your OHLCV data
df = pd.read_parquet("path/to/OHLCV1D.parquet")

# Create and train generator
generator = SyntheticOHLCVGenerator(verbose=True)
generator.fit("SYMBOL", df)

# Generate scenarios
scenarios = generator.sample(
    seeds=[1, 2, 3],
    steps=252,
    last_history_dt=df.index[-1]
)

🛠️ Configuration

Environment Setup

The package uses structured configuration classes:

from syntheticdata import GeneratorConfig, DataConfig, SamplingConfig
import pandas as pd

# Generator configuration
gen_config = GeneratorConfig(
    verbose=True,
    epochs=128,
    sample_size=1
)

# Data configuration
data_config = DataConfig(
    root_folder="./data",
    symbols=["ACB", "VCB"],
    startdate=pd.Timestamp("2020-01-01"),
    stopdate=pd.Timestamp("2024-01-01")
)

# Sampling configuration
sampling_config = SamplingConfig(
    seeds=[1, 2, 3],
    steps=252,
    save=True,
    output_format="parquet"
)

Model Storage

Models are stored in organized ticker folders:

data_folder/
├── SYMBOL/
│   ├── models/
│   │   └── SYMBOL_YYYY-MM-DD/    # Model files
│   │       ├── synthesizer.pkl
│   │       ├── state.json
│   │       └── version.json
│   ├── syntheticdata/            # Generated scenarios
│   └── OHLCV1D.parquet          # Original data

🧪 Testing

Run the comprehensive test suite:

# Activate virtual environment first
.venv-synthetic\Scripts\activate

# Run enhanced test suite
python test_enhanced.py

# Run specific tests
python -m pytest tests/ -v

Expected output:

Enhanced SyntheticData Testing Suite
====================================
✅ Configuration tests PASSED
✅ Enhanced API tests PASSED  
✅ Quick functions tests PASSED
🎉 ALL TESTS PASSED!

📖 Usage Examples

Example 1: Train and Generate for Single Symbol

from syntheticdata import SyntheticDataAPI

# Initialize
api = SyntheticDataAPI()

# Check available symbols
symbols = api.get_available_symbols("./data")
print(f"Available symbols: {symbols}")

# Train model
result = api.train_single_model(
    symbol="ACB",
    data_folder="./data",
    start_date="2020-01-01", 
    end_date="2024-01-01"
)

if result['status'] == 'success':
    print(f"Model trained! Data points: {result['data_points']}")
    
    # Generate scenarios
    scenarios = api.generate_single_scenario(
        symbol="ACB",
        data_folder="./data",
        seeds=[1, 2, 3, 4, 5],
        steps=252,
        save_results=True,
        output_format="parquet"
    )
    
    print(f"Generated {len(scenarios['scenarios'])} scenarios")

Example 2: Batch Processing Multiple Symbols

from syntheticdata import SyntheticDataAPI

api = SyntheticDataAPI()

symbols = ["ACB", "VCB", "TCB", "MBB"]  # Bank stocks
results = api.batch_generate(
    symbols=symbols,
    data_folder="./data",
    seeds=[1, 2, 3],
    steps=252,
    save_results=True
)

for symbol, result in results.items():
    if result['status'] == 'success':
        print(f"{symbol}: {len(result['scenarios'])} scenarios generated")
    else:
        print(f"{symbol}: Error - {result['error']}")

Example 3: Check Model Status

from syntheticdata import SyntheticDataAPI

api = SyntheticDataAPI()

# Check if model exists and get info
status = api.check_model_status(
    symbol="ACB",
    data_folder="./data",
    end_date="2024-01-01"
)

print(f"Status: {status['status']}")
if status['status'] == 'found':
    print(f"Model path: {status['model_path']}")
    print(f"Last close: ${status['last_close']:.2f}")
    print(f"SDV version: {status['sdv_version']}")

🔧 CLI Usage

The package includes a command-line interface:

# Show help
python -m syntheticdata --help

# Train a model
python -m syntheticdata train --symbol ACB --data-folder ./data

# Generate scenarios  
python -m syntheticdata generate --symbol ACB --data-folder ./data --seeds 1,2,3 --steps 252

# Batch processing
python -m syntheticdata batch --symbols ACB,VCB,TCB --data-folder ./data

📋 Requirements

• Python 3.8+
• SDV 1.27.0
• pandas >= 1.5.0
• numpy >= 1.20.0
• torch >= 1.10.0
• Additional dependencies in requirements.txt

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests (python test_enhanced.py)
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Troubleshooting

Common Issues

1. ModuleNotFoundError: Make sure you've installed the package with pip install -e .

2. Model not found: Ensure you're using consistent date parameters for training and status checking

3. SDV warnings: Future warnings from SDV are expected and don't affect functionality

4. Memory issues: For large datasets, consider reducing epochs or sample_size in GeneratorConfig

Getting Help

• Check the examples/ directory for working code samples
• Run python test_enhanced.py to validate your installation
• Review the comprehensive test suite for usage patterns

🔮 Roadmap

• Additional synthetic data models beyond PARSynthesizer
• Web interface for easy model management
• Integration with popular trading platforms
• Advanced backtesting capabilities
• Docker containerization

---

Made with ❤️ for the quantitative finance community