Option to overwrite existing directory

[gibsramen]

I'd like to use Empress in a Snakemake pipeline (see below) but I'm running into an annoying issue. If my understanding is correct (which it could very well not be), the way Snakemake works is that it sees that the results/empress directory does not exist and thus creates it before the shell command executes. However, when this occurs Empress complains that Output directory already exists!. I think what would be useful is for some sort of --overwrite option to force the creation of the visualization + support files.

In my case, I can work around this by changing some of the filepaths but I think this might be a useful option.

rule empress:
    input:
        tree="results/diversity/metagenomics/phylo_rpca/labeled-phylogeny.nwk",
        feature_md="results/tmp/feature_metadata.tmp.tsv"
    output:
        "results/empress/empress.html"
    shell:
        """
        empress tree-plot \
            --tree {input.tree} \
            --feature-metadata {input.feature_md} \
            --output-dir results/empress
        """

[fedarko]

Eesh, that's annoying! It should be feasible to add this in. Looks like the lines causing the problem are here:

empress/empress/_plot_utils.py

Lines 86 to 101 in e543610

	def check_and_process_files(output_dir, tree_file, feature_metadata):
	"""Initial checks and processing of files for standalone CLI plotting.
	Parameters
	----------
	output_dir : str
	tree_file : str
	fm_file : str
	Returns
	-------
	bp.Tree
	pd.DataFrame
	"""
	if os.path.isdir(output_dir):
	raise OSError("Output directory already exists!")

... So we can probably just add an --overwrite / --no-overwrite flag (default --no-overwrite, i.e. the current behavior) to the standalone script, and then pass the value of this flag directly to check_and_process_files() to tell it to ignore the output directory already existing.

(Maybe we should add some code to check_and_process_files() to, when overwrite is True, just delete the contents of the output directory? This would limit potential complications with stuff already existing in the output directory, although it might be dangerous if people use this willy-nilly. Not sure what the best practice is.)

[gibsramen]

I think deleting the contents of the output directory might be too dangerous (who knows what other stuff people could have in there). If we could only overwrite empress.html and support_files/* I think that would be sufficient.

Also I wrote this code so it's my own fault this is an annoying issue 😅

[ElDeveloper]

Overwriting is definitely the way to go. This used to be the behavior of `make_emperor.py` (the predecessor to q2-emperor) and this approached worked great for years. From: Gibs ***@***.***> Date: Saturday, May 21, 2022 at 6:46 AM To: biocore/empress ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [biocore/empress] Option to overwrite existing directory (Issue #558) I think deleting the contents of the output directory might be too dangerous (who knows what other stuff people could have in there). If we could only overwrite empress.html and support_files/* I think that would be sufficient. Also I wrote this code so it's my own fault this is an annoying issue 😅 — Reply to this email directly, view it on GitHub<#558 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC3UC2SO3SINSZ6CV7JOW3VLDSM5ANCNFSM5WP54DJQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[gibsramen]

So this specific instance can be circumvented with

output:
    directory("results/empress")

but I think it's still worth adding the --overwrite option. Will try to carve out some time to finish up my PR.