To use the distributed training and evaluation scripts, ensure the following Python packages are installed:
- torch (>=2.7.0)
- torchvision
- torchaudio
- cudatoolkit (if using GPU)
- datasets
- transformers
- tokenizers
- accelerate
- wandb
- evaluate
- ray[data,train,tune,serve]
- peft
- tqdm
- scikit-learn
- s3fs
You can install the core requirements with:
pip install -q torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -q transformers datasets accelerate wandb evaluate tqdm scikit-learn "ray[default]" peft boto3 s3fsRay enables distributed training and evaluation across multiple nodes or GPUs. You can use Ray both locally (on a single machine) or on a cluster (e.g., AWS, using a YAML config).
From a YAML config (cloud/cluster):
ray up <path-to-ray-cluster-yaml>This will start the Ray head and worker nodes as defined in your YAML file.
Locally (single machine):
ray start --head --port=6379Using ray submit (for remote clusters):
ray submit <path-to-ray-cluster-yaml> scripts/<your_script>.py -- <script-args>- The
--separates Ray arguments from your script's arguments. - Example:
ray submit <path-to-ray-cluster-yaml> scripts/ray_train.py -- --dataset <DATASET_PATH> --model lstm --use-ray --num_workers 4Using ray job submit (Ray 2.x+):
If your cluster is already running, you can submit a job from your local machine:
ray job submit --address='ray://<head-node-ip>:10001' -- python scripts/<your_script>.py <script-args>- Ray Dashboard:
- Access the Ray dashboard at
http://<head-node-ip>:8265(or the port specified in your cluster config). - If using AWS SSO login and SSM:
- You can port forward the dashboard using AWS SSM:
aws ssm start-session \ --target <instance-id> \ --document-name AWS-StartPortForwardingSession \ --parameters '{"portNumber":["8265"], "localPortNumber":["8265"]}'
- Replace
<instance-id>with your Ray head node's EC2 instance ID. This will forward the remote dashboard port 8265 to your local port 8265.
- You can port forward the dashboard using AWS SSM:
- If using SSH:
- You can use SSH port forwarding if needed:
ssh -L 8265:localhost:8265 <ssh-username>@<head-node-ip>
- You can use SSH port forwarding if needed:
- Access the Ray dashboard at
- Command-line:
- Check cluster status:
ray status
- List running jobs:
ray job list
- Check cluster status:
ray down <path-to-ray-cluster-yaml>This script performs distributed evaluation of a pre-trained Masked Language Model (MLM) using Ray.
Example usage:
python scripts/ray_evaluator.py \
--raw_dataset_dir <RAW_DATASET_DIR> \
--model_name <MODEL_NAME_OR_PATH> \
--batch_size 128 \
--mlm_prob 0.15 \
--wandb_project <WANDB_PROJECT_NAME>--raw_dataset_dir: Path to the raw HuggingFace dataset directory.--model_name: Model name or path (e.g., 'Rostlab/prot_bert_bfd').--batch_size: Batch size for evaluation.--mlm_prob: Masked LM probability.--wandb_project: (Optional) W&B project name for logging.--schema_path: (Optional) Path to a JSON file with masking schema.--test: (Optional) Run on a small subset for testing.
This script performs distributed fine-tuning of a pre-trained MLM or custom model using Ray and HuggingFace Trainer.
Example usage:
python scripts/ray_fine_tune.py \
--config <CONFIG_JSON_PATH> \
--num_workers 8 \
--s3_output_path <S3_OUTPUT_PATH>--config: Path to a JSON config file (see template below).--num_workers: Number of Ray workers (GPUs).--s3_output_path: (Optional) S3 path to sync the best model after training.--test: (Optional) Run in test mode with a smaller dataset.
This script trains custom LSTM/Transformer models using Ray for distributed training.
Example usage:
python scripts/ray_train.py \
--dataset <DATASET_PATH> \
--model <MODEL_TYPE> \
--tokenizer <TOKENIZER_PATH> \
--use-ray \
--num_workers 4--dataset: Path to the HuggingFace dataset.--model: Model type ('lstm', 'bilstm', 'transformer').--tokenizer: Path to the tokenizer file.--use-ray: Enable Ray distributed training.--num_workers: Number of Ray workers (GPUs).
This script processes raw Parquet files and creates tokenized datasets for training. It supports both HuggingFace models (BERT, ProtBERT, ESM, LLaMA) and custom models (LSTM, Transformer).
Example usage (local):
# Process data for HuggingFace models (BERT, ProtBERT, ESM, LLaMA)
python scripts/ray_datawriter.py \
--path "data/*.parquet" \
--model-name protbert \
--output-raw "processed_data" \
--max-len 1024 \
--use-ray \
--num-workers 4
# Process data for custom models (LSTM, Transformer)
python scripts/ray_datawriter.py \
--path "data/*.parquet" \
--model-name lstm \
--output-raw "processed_data" \
--max-len 1024 \
--bpe-vocab 200 \
--truncate-long \
--use-ray \
--num-workers 4Example usage (AWS with Ray cluster):
# Submit to Ray cluster
ray submit <path-to-ray-cluster-yaml> scripts/ray_datawriter.py -- \
--path "s3://your-bucket/data/*.parquet" \
--model-name protbert \
--output-raw "s3://your-bucket/processed_data" \
--max-len 1024 \
--use-ray \
--num-workers 8 \
--s3-key <your-aws-key> \
--s3-secret <your-aws-secret>
# Or using ray job submit
ray job submit --address='ray://<head-node-ip>:10001' -- \
python scripts/ray_datawriter.py \
--path "s3://your-bucket/data/*.parquet" \
--model-name lstm \
--output-raw "s3://your-bucket/processed_data" \
--max-len 1024 \
--bpe-vocab 200 \
--truncate-long \
--use-ray \
--num-workers 8 \
--s3-key <your-aws-key> \
--s3-secret <your-aws-secret>Key parameters:
--path: Path to Parquet files (supports glob patterns and S3 paths)--model-name: Model type ('bert', 'protbert', 'esm', 'llama', 'lstm', 'transformer')--output-raw: Output directory for processed dataset--max-len: Maximum sequence length (default: 1024)--bpe-vocab: Vocabulary size for BPE tokenizer (custom models only, default: 200)--truncate-long: Truncate long sequences instead of padding--use-ray: Enable Ray distributed processing--num-workers: Number of Ray workers--s3-key,--s3-secret: AWS credentials for S3 access
You can use Ray for orchestration on a single machine without launching a distributed cluster. This is useful for debugging or running on a single GPU/CPU, but still benefits from Ray's job management and logging.
Example: Run with a single Ray worker (local, non-distributed):
python scripts/ray_train.py \
--dataset <DATASET_PATH> \
--model <MODEL_TYPE> \
--tokenizer <TOKENIZER_PATH> \
--use-ray \
--num_workers 1Or for fine-tuning:
python scripts/ray_fine_tune.py \
--config <CONFIG_JSON_PATH> \
--num_workers 1Or for data processing:
python scripts/ray_datawriter.py \
--path "data/*.parquet" \
--model-name protbert \
--output-raw "processed_data" \
--max-len 1024 \
--use-ray \
--num-workers 1- The
--use-rayand--num_workers 1flags ensure Ray is used, but only a single worker/process is launched. - No Ray cluster or YAML config is needed for this mode; Ray will run locally.
Below is a template for a Ray cluster configuration YAML file for AWS, based on a real working example. Replace all placeholder values (in angle brackets) with your actual configuration. Do not use real secrets or tokens in your config.
# ray_cluster_template.yaml
cluster_name: <your-cluster-name>
provider:
type: aws
region: <aws-region>
aws_profile: <aws-profile-name>
use_internal_ips: true
auth:
ssh_user: <ssh-username>
ssh_proxy_command: |
aws ssm start-session --target `aws ec2 describe-instances --filters 'Name=tag:Name,Values=<head-node-tag>' 'Name=instance-state-name,Values=running' --query 'Reservations[*].Instances[*].InstanceId' --output text | head -n1` --document-name AWS-StartSSHSession --parameters 'portNumber=22'
available_node_types:
ray.head.default:
node_config:
InstanceType: <head-instance-type>
ImageId: <ami-id>
SubnetId: <subnet-id>
KeyName: <key-name>
IamInstanceProfile:
Arn: <iam-instance-profile-arn>
InstanceMarketOptions:
MarketType: spot
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 100
VolumeType: gp3
DeleteOnTermination: true
ray.worker.default:
min_workers: <min-workers>
max_workers: <max-workers>
node_config:
InstanceType: <worker-instance-type>
ImageId: <ami-id>
SubnetId: <subnet-id>
KeyName: <key-name>
IamInstanceProfile:
Arn: <iam-instance-profile-arn>
InstanceMarketOptions:
MarketType: spot
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 100
VolumeType: gp3
DeleteOnTermination: true
head_node_type: ray.head.default
max_workers: <max-workers>
setup_commands:
- |
if ! command -v python &> /dev/null; then
sudo ln -s $(which python3) /usr/bin/python
fi
# Format and mount NVMe to /scratch if not already formatted
- |
sudo chown <ssh-username>:<ssh-username> /opt/dlami/nvme
export HF_CACHE=/opt/dlami/nvme/hf_cache
export PATH=/home/<ssh-username>/.local/bin:$PATH
# Install git if needed
- |
if ! command -v git &> /dev/null; then
sudo apt update
sudo apt install -y git
fi
# Clone your repo if not already present
- |
if [ ! -d "/home/<ssh-username>/<repo-name>" ]; then
git clone <your-repo-url> /home/<ssh-username>/<repo-name>
fi
- |
cd /home/<ssh-username>/<repo-name>
ln -s /opt/dlami/nvme data
# Install AWS CLI if needed
- |
if ! command -v aws &> /dev/null; then
sudo apt update
sudo apt install -y awscli
fi
# Copy data from S3 to /scratch (optional)
# - aws s3 cp --recursive <s3-bucket-path> /home/<ssh-username>/<repo-name>/data/<dataset-dir> --quiet
- pip install -q --upgrade pip
- pip install -q torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
- pip install -q transformers datasets accelerate wandb evaluate tqdm scikit-learn s3fs tokenizers "ray[data,train,tune,serve]" peft
# wandb login (optional)
# - wandb login <your-wandb-api-key>
file_mounts: {}
initialization_commands: []Below is a template for the fine-tuning configuration JSON file used by ray_fine_tune.py. Replace all placeholder values (in angle brackets) with your actual configuration.
{
"model_key": "protbert",
"dataset_path": "<path-to-hf-dataset>",
"checkpoint_path": "<output-checkpoint-dir>",
"num_epochs": 3,
"batch_size": 32,
"learning_rate": 0.0001,
"mlm_prob": 0.15,
"wandb_project": "<wandb-project-name>",
"tokenizer_path": "<path-to-tokenizer>",
"gradient_checkpointing": true,
"use_lora": false,
"lora_r": 16,
"lora_alpha": 32,
"lora_target_modules": ["query", "value"],
"lora_dropout": 0.05,
"early_stop": true,
"early_stopping_patience": 3,
"s3_output_path": "<s3-bucket-path>"
}