MiniPhy – Minimization via Phylogenetic compression (former MOF-Compress)

Workflow for phylogenetic compression of microbial genomes, producing highly compressed .tar.xz genome archives. MiniPhy first estimates the evolutionary history of user-provided genomes and then uses it for guiding their compression using XZ. The resulting archives can be distributed to users or re-compressed/indexed by other methods. For more information, see the website of phylogenetic compression and the associated paper.

Contents

• 1. Introduction
• 2. Dependencies
  • 2a. Essential dependencies
  • 2b. Protocol-specific dependencies
• 3. Installation
• 4. Usage
  • 4a. Basic example
  • 4b. Adjusting configuration
  • 4c. List of implemented protocols
  • 4d. List of workflow commands
  • 4e. Running on a cluster
  • 4f. Troubleshooting
• 5. Citation
• 6. Issues
• 7. Changelog
• 8. License
• 9. Contacts

1. Introduction

The user provides files of files for individual batches in the input/ directory and specifies the requested compression protocols in the configuration file. It is assumed that the input genomes are provided as batches of phylogenetically related genomes, of up to approx. 10k genomes per batch (for more information on batching strategies, see the paper). Upon the execution by make, MiniPhy performs phylogenetic compression of the assemblies or associated de Bruijn graphs. All the compressed outputs and the calculated statistics are then placed in output/.

2. Dependencies

2a. Essential dependencies

• Conda (unless the use of Conda is switched off in the configuration) and ideally also Mamba (>= 0.20.0)
• GNU Make
• Python (>=3.7)
• Snakemake (>=6.2.0)
• XZ

and can be installed by Conda by

conda install -c conda-forge -c bioconda -c defaults \
  make "python>=3.7" "snakemake-minimal>=6.2.0" "mamba>=0.20.0"

2b. Protocol-specific dependencies

These are installed automatically by Snakemake when they are requested; for instance, ProPhyle is not installed unless Protocol 3 is used. The specifications of individual environments can be found in workflow/envs/, and they contain: Attotree, ETE 3, SeqTK, xopen, Pandas, Jellyfish 2, ProphAsm, and ProPhyle.

All non-essential dependencies across all protocols can also be installed at once by make conda.

3. Installation

Clone and enter the repository by

git clone https://github.com/karel-brinda/miniphy
cd miniphy

Alternatively, the repository can also be installed using cURL by

mkdir miniphy
cd miniphy
curl -L https://github.com/karel-brinda/miniphy/tarball/main \
    | tar xvf - --strip-components=1

4. Usage

4a. Basic example

• Step 1: Provide lists of input files.
  For every batch, create a txt list of input files in the input/ directory (i.e., as input/{batch_name}.txt. Use either absolute paths (recommended), or paths relative to the root of the GitHub repository (not relative to the txt files).

  Such a list can be generated, for instance, by find by

  find ~/dir_with_my_genomes -name '*.fa' > input/my_first_batch.txt

  Alternatively, if you have a tab-separated metadata file with one genome per row, you can generate MiniPhy batch lists with create_batches.py. The default metadata columns are species and filename:

  ./create_batches.py meta_file.tsv -d input

  This creates one or more input/{batch_name}.txt files. Each file contains paths to genomes that will be compressed together as one MiniPhy batch.

  For metadata files with different column names, use -s for the species column and -f for the genome filename or path column:

  ./create_batches.py meta_file.tsv.xz \
    -s hit1_species \
    -f asm_path \
    -d input

  The script groups genomes by cleaned species names. Species clusters smaller than -m are moved to the dustbin; species clusters larger than -M are split into several batches; dustbin batches are split using -D.

  For example:

  ./create_batches.py meta_file.tsv.xz \
    -s hit1_species \
    -f asm_path \
    -m 100 \
    -M 4000 \
    -D 1000 \
    -d input

  To avoid mixing old and new batch lists, create_batches.py refuses to write into an output directory that already contains .txt files. Remove old batch lists manually before rerunning, or use --force to delete existing .txt files in the selected output directory before generating new ones:

  ./create_batches.py meta_file.tsv.xz \
    -s hit1_species \
    -f asm_path \
    -d input \
    --force

  Use --force carefully: it removes all existing .txt files in the selected output directory.

  The supported input file formats include FASTA and FASTQ (possibly compressed by GZip).

• Step 2 (optional): Provide corresponding phylogenies.
  Instead of estimating phylogenies by Attotree (similar functionality like Mashtree), it is possible to supply custom phylogenies in the Newick format. The tree files should be named input/{batch_name}.nw, and the leaf names inside should correspond to FASTA filenames (without FASTA suffixes).

• Step 3 (optional): Adjust configuration.
  By editing config.yaml it is possible to specify compression protocols, data analyses, and low-level parameters (see below).

• Step 4: Run the pipeline.
  Run the pipeline by make; this is run Snakemake with the corresponding parameters.

• Step 5: Retrieve the output files.
  All output files will be located in output/.

For larger collections, the usual workflow is:

1. prepare a metadata table with one row per genome;
2. use create_batches.py to create input/{batch_name}.txt files;
3. optionally add matching input/{batch_name}.nw Newick trees for batches where a custom phylogeny should be used;
4. edit config.yaml to select the requested protocols and resource settings;
5. run make.

For large collections, prefer absolute genome paths in the generated batch files. This makes the batch lists independent of where MiniPhy is launched from.

4b. Adjusting configuration

The workflow can be configured via the config.yaml file, and all options are documented directly there. The configurable functionality includes:

• switching off Conda,
• protocols to use (asm, dGSs, dBGs with propagation),
• analyzes to include (sequence and k-mer statistics),
• k for de Bruijn graph and k-mer counting,
• Attotree parameters (phylogeny estimation),
• XZ parameters (low-level compression), or
• JellyFish parameters (k-mer counting).

4c. List of implemented protocols

Protocol	Representation	Description	Product
Protocol 1 (default)	Assemblies	Left-to-right reordering of the assemblies according to the phylogeny	output/asm/{batch}.tar.xz original assemblies in FASTA (1)
Protocol 2 (optional)	de Bruijn graphs	Simplitigs from individual assemblies, left-to-right reordering of their files	output/pre/{batch}.tar.xz with simplitig text files, representing individual de Bruijn graphs
Protocol 3 (optional)	de Bruijn graphs	Bottom-up k-mer propagation using ProPhyle, simplitigs at individual nodes of the tree, and left-to-right re-ordering of the obtained files	output/post/{batch}.tar.xz output/post/{batch}.nw simplitig text files per individual nodes of the tree (2)

⁽¹⁾ In FASTA 1-line format and all sequences converted to uppercase (unless switched off in the configuration).
⁽²⁾ The original de Bruijn graphs can be obtained by merging k-mer sets along the respective root-to-leaf paths.

4d. List of workflow commands

MiniPhy is executed via GNU Make, which handles all parameters and passes them to Snakemake. Here's a list of all implemented commands (to be executed as make {command}):

######################
## General commands ##
######################
    all                  Run everything (the default subcommand)
    help                 Print help messages
    conda                Create the conda environments
    clean                Clean all output archives and files with statistics
    cleanall             Clean everything but Conda, Snakemake, and input files
    cleanallall          Clean completely everything
###############
## Reporting ##
###############
    viewconf             View configuration without comments
    reports              Create html report
####################
## For developers ##
####################
    test                 Run the workflow on test data (P1)
    bigtest              Run the workflow on test data (P1, P2, P3)
    format               Reformat all source code
    checkformat          Check source code format

Note: make format and make checkformat require YAPF and Snakefmt, which can be installed by conda install -c conda-forge -bioconda yapf snakefmt.

4e. Running on a cluster

Cluster-related parameters for Snakemake can be added via the SMK_CLUSTER_ARGS Make variable.

Example:

make SMK_CLUSTER_ARGS="--profile my_snakemake_cluster_profile"

4f. Troubleshooting

Tests can be run by make test (just Protocol 1) or make bigtest (all the protocols).

Protocol 3 uses ProPhyle-based k-mer propagation and has additional compiled dependencies. If Protocol 3 fails during environment creation or compilation, first check that the Conda environments were created correctly:

make conda

If the problem is specific to Protocol 3 and you only need assembly compression, disable Protocol 3 in config.yaml:

protocol_post: False

The default Protocol 1 does not require Protocol 3.

5. Citation

K. Břinda, L. Lima, S. Pignotti, N. Quinones-Olvera, K. Salikhov, R. Chikhi, G. Kucherov, Z. Iqbal, and M. Baym. Efficient and robust search of microbial genomes via phylogenetic compression. Nature Methods 22, 692–697 (2025). https://doi.org/10.1038/s41592-025-02625-2 (PDF)

@article {PhylogeneticCompression,
   author  = {Karel B{\v r}inda and Leandro Lima and Simone Pignotti
               and Natalia Quinones-Olvera and Kamil Salikhov and Rayan Chikhi
               and Gregory Kucherov and Zamin Iqbal and Michael Baym},
   title   = {Efficient and robust search of microbial genomes via phylogenetic compression},
   journal = {Nature Methods},
   volume  = {22},
   pages   = {692--697},
   year    = {2025},
   doi     = {10.1038/s41592-025-02625-2},
   url     = {https://www.nature.com/articles/s41592-025-02625-2}
}

6. Issues

Please use GitHub issues.

7. Changelog

See Releases.

8. License

MIT

9. Contacts

• Karel Brinda <karel.brinda@inria.fr>
• Leandro Lima <leandro@ebi.ac.uk>