PRISM: Multimodal Continual Instruction Tuning Toolbox

A plug-in, reproducible infrastructure for scalable multimodal continual instruction tuning.

🔥 News • 📖 Introduction • 🧩 Methods • 🚀 How To Use • 📝 Citation • 📄 License • 📧 Contact

PRISM is a plug-in, reproducible toolbox for training and evaluating multimodal large language models (MLLMs) under multimodal continual instruction tuning (MCIT). A single entry point (run.py) orchestrates sequential task training, inference, and evaluation across multiple benchmarks and continual-learning methods.

---

🔥 News

• [2026-08] 🎉 CRAM is accepted to EMNLP 2026! Paper · Implementation is available in PRISM via --method cram.
• [2026-06] 🚀 ProtoAda is released! Check out ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning.
• [2026-05] 🚀 PRISM is released! Our plug-in infrastructure currently supports 3 MCIT benchmarks, 12 continual-learning methods, and a zero-shot baseline. Paper · Documentation
• [2026-05] 🚀 DRAPE is released! Check out DRAPE: Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning.
• [2026-04] 🎉 SAME is accepted to ICML 2026! Paper · Implementation is available in PRISM via --method same.

---

✨ Highlights

• Plug-and-play methods. Add a continual-learning strategy as an independent plugin without modifying the underlying MLLM codebase.
• Unified experimentation. Train, infer, and evaluate sequential tasks through a single entry point with standardized configurations and checkpoint layouts.
• Broad MCIT coverage. Support representative rehearsal, regularization, prompt-based, LoRA-based, and Mixture-of-Experts approaches under one infrastructure.
• Easy extensibility. New methods, benchmarks, and backbones can be registered with lightweight interfaces for fast prototyping and fair comparison.

---

📖 Introduction

Multimodal large language models (MLLMs) unify diverse vision and vision–language tasks into a shared instruction-following format. In real deployments, however, data and instructions arrive as streams: models must learn new tasks sequentially without erasing earlier capabilities. Standard fine-tuning suffers from catastrophic forgetting under this setting.

Multimodal continual instruction tuning (MCIT) addresses this by training MLLMs on a sequence of instruction-tuning stages while preserving performance on prior tasks. PRISM standardizes this workflow—benchmark definitions, method integrations, checkpoint layout, and evaluation—so that MCIT methods can be compared and extended under one infrastructure.

---

🧩 Methods Implemented

Each method is selected with --method <id> (folder under method/custom/<id>/).

Abbr.	--method	Paper
HiDe-LLaVA	hide_llava	HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
Replay+LoRA	replay_lora	LoRA: Low-Rank Adaptation of Large Language Models
LoRA	ft_lora	LoRA: Low-Rank Adaptation of Large Language Models
O-LoRA	olora	Orthogonal Subspace Learning for Language Model Continual Learning
SMoLoRA	smolora	SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
MoELoRA	moelora	CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
CL-MoE	clmoe	CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
ModalPrompt	modal_prompt	ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
EWC	ewc	Overcoming catastrophic forgetting in neural networks
DisCo	disco	Federated Continual Instruction Tuning
SAME	same	SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
CRAM	cram	CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Zero-shot	zeroshot	Visual Instruction Tuning

💡 Adding a new method? Implement method/custom/<your_method>/integration.py and register it with @CLMethodFactory.register("your_method").

---

🚀 How To Use

1. Environment Setup

If you are on NVIDIA RTX 5090 GPU(s) (our tested setup), a single command sets up everything from the repository root:

bash scripts/setup_env.sh

This creates the conda environment prism (if missing), installs torch 2.8 + cu128, training/evaluation dependencies, flash-attn, and runs pip install -e ..

For other GPUs or CUDA versions, you may need to adjust PyTorch, flash-attn, and related libraries. See requirements/README.md for options (e.g., TORCH_REQUIREMENTS=requirements/torch-cu118.txt for older CUDA stacks, FLASH_ATTN_WHEEL, and SKIP_FLASH_ATTN).

Activate and verify:

conda activate prism
python -c "import torch; import transformers; import deepspeed; print(torch.__version__, transformers.__version__)"

2. Pre-trained Weights

Download weights from each repository's Model Zoo, then set paths in config/paths/llava_paths.py or config/paths/internvl_paths.py. Select the backbone via backbone in config/run_config.py (llava or internvl).

• LLaVA — llava-v1.5-7b
• InternVL — InternVL-Chat-ViT-6B-Vicuna-7B

You can plug in additional backbones under config/backbone/ and backbone/, then register them in config/backbone/registry.py.

3. Datasets

PRISM currently supports three benchmarks:

Benchmark	--benchmark	Tasks	Reference
CoIN	coin	8	Paper · Benchmark
UCIT	ucit	6	Paper · Benchmark
TriGap	trigap	10	Paper · Benchmark

A benchmark typically has an image folder and an instruction folder. JSON files in the instruction folder reference image paths, so your on-disk layout must match those paths.

Set benchmark paths in config/benchmarks/<benchmark>.py (e.g., TRIGAP_IMAGE_DIR and TRIGAP_INSTRUCTION_DIR in TriGap.py).

For quick experiments, you can use smaller sub-splits: sample the instruction JSON yourself, save it with a _sub suffix (e.g., train_sub.json), and set "use_sub_dataset": true in config/run_config.py.

You can add custom benchmarks under config/benchmarks/ and register them in config/benchmarks/__init__.py.

4. Paths and Config

Edit backbone paths under config/paths/ and benchmark roots under config/benchmarks/. Tune runs via config/run_config.py.

5. Run

First, run a quick zero-shot inference on a single task to check weights, data paths, and GPUs (zeroshot uses the base MLLM checkpoint only):

python run.py infer 0 --method zeroshot

Then run continual training and evaluation:

python run.py train 0 1 2
python run.py infer 0 1 2

0, 1, and 2 are task indices (see config/benchmarks/<benchmark>.py). You may train any tasks you need; stage k resumes from task k−1's checkpoint. For inference, choose the checkpoint in config/run_config.py.

CLI flags override config; omitted flags use config defaults.

---

📝 Citation

If PRISM or the implemented methods are useful for your research, please consider citing the corresponding papers:

@article{tang2026prism,
  title={Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning},
  author={Jun-Tao Tang and Yu-Cheng Shi and Zhen-Hao Xie and Da-Wei Zhou},
  year={2026},
  journal={arXiv preprint arXiv:2605.26110},
}

@inproceedings{xie2026same,
  title={SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning},
  author={Xie, Zhen-Hao and Tang, Jun-Tao and Shi, Yu-Cheng and Ye, Han-Jia and Zhan, De-Chuan and Zhou, Da-Wei},
  booktitle={ICML},
  year={2026}
}

@inproceedings{tang2026cram,
  title={CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning},
  author={Tang, Jun-Tao and Xie, Zhen-Hao and Shi, Yu-Cheng and Zhou, Da-Wei},
  booktitle={EMNLP},
  year={2026}
}

---

📄 License

This project is released under the MIT License.

---

🙏 Acknowledgments

We thank the following projects for their benchmarks and reference implementations used in PRISM:

• UCIT
• CoIN
• MCITlib

---

📧 Contact

If you have any questions, suggestions, or feature requests, please open an issue or contact the authors: Jun-Tao Tang (juntao.tang@smail.nju.edu.cn), Yu-Cheng Shi (231250034@smail.nju.edu.cn), and Da-Wei Zhou (zhoudw@lamda.nju.edu.cn).

If you find PRISM useful, a ⭐ on GitHub is always appreciated!