Flye crash in the workflow "genomeassembler"

[BrunoBertolasi]

Hello to all,

I am currently using the workflow genomeassembler (that incorporate Flye) (https://nf-co.re/genomeassembler/dev/) to assemble my 12,5 GB of reads obtained form a ONT MinIon sequencing. My PC is a 16 cpu cores with 125 Gb of RAM.

The elaboration started with no problems, but it stopped at a certain point, when Flye had already performed some tasks yet. I attach the genomeassembler pipeline logfile nextflow.log and the specific Flye logfile I found in the partial results folder flye.log.

Despite the failure it seems the process have produced some results, given that, in the shell screen, there are some information about genome size (around 0,7 Gbp as expected) and some files are produced in the workflow working directories.

Taking in to the account I can pass additional arguments about Flye in the main workflow command is there any setting of Flye that could solve the problem? My only actual Flye configuration, besides the default ones used in the genomeassembler workflow, is --min-overlap 3000.

Any suggestions are welcome.

Bruno

[mikolmogorov]

Hi Bruno,

The log suggest that Flye binary crashed at the repeat graph stage, but it is unclear why. Could you try running Flye on the same input without Nexftwol (e.g. as a bioconda package) to check if it works? You can also use the same output directory and restart from repeat stage using --resume.

[BrunoBertolasi]

Hi Mikhail,

I installed a stand alone version of Flye and I created a directory "Flye", along the ones used by the original pipeline. I then run Flye stand alone, as suggested,using the same setting I deducted from the .log file of the previous crash.
Unluckily the process crashed again after some time. I attached the latest .log file produced flye.log

Furthermore can you please explain to me how to use the option -resume, if eventually I will be able to unblock the pipeline flow?

Thanks for your support!

Bruno

[mikolmogorov]

Thanks! Looks like a bug.. To fix it, I'll need to reproduce it on my end, is it possible for you to share the flye/10-consensus/consensus.fasta file? No need to share the reads.

[BrunoBertolasi]

Thanks Mikhail,
I'd glad to share the required file. I uploaded it in a Google Drive folder that I could share with you if you want. Otherwise let me know your preferred option.
Bye!

[mikolmogorov]

Thanks - please send the link to mikolmogorov@gmail.com

[BrunoBertolasi]

Thanks Mikhail,
You should have received a mail with the link to Google Drive folder with the file.

[mikolmogorov]

@BrunoBertolasi thanks for sending the assembly! It's strange, but I was not able to reproduce the problem on my end - it goes though the stage which crashed for you without a problem. Could you share how you installed Flye? If it's a binary release, I'd try to compile from source and try restarting the repeat stage. What is the OS version? If you have access to a different machine by chance, I'd give it a try too.

[BrunoBertolasi]

Hi Mikhail,

I installed Flye from source with the command python3 setup.py install as suggested in the installation paragraph. My PC is a 16 cpu core with 128 Gb of RAM and the OS is Ubuntu 22.04.5 LTS.

In the meaning time I tried another run using less processors (this time 12 out 16; in the previous run I used all 16 cpu) and, reading the log file flye.log, I noted that the process stopped at an earlier stage than the run performed with all 16 processors. Is then possible that the failure is caused by lack of computational power?

[mikolmogorov]

I think it could be that you are running out of memory or disk space. Do you have any additional messages in terminal output? Or anything relevant if you run dmesg?

[BrunoBertolasi]

Hi Mikhail,

I currently have about 488 Gb of free space in my HD. About RAM, during Flye process, I have always noted that only about a quarter of my 128 Gb is used so I suppose the memory is OK.

About the output of the kernel check I am not able to evaluate it but I attach here the file dmesg.txt

[mikolmogorov]

1. Does Flye works on other datasets on your machine? e.g. a small test case, and also something of comparable genome size?
2. Could you post a complete terminal output when Flye does crash (not the log file).

[BrunoBertolasi]

Hello Mikhail,

I attach the last flye.log along with the correspondent Terminal output.txt relative to my data.

As you suggested, I tried a Flye run with small mitochondrial genome (958,3 MB of ONT reads in .fastq.gz format) and the program completed successfully the task until the end.

What are your considerations?

[mikolmogorov]

Thank you - these new logs seem to show that now Flye crash on a completely different stage (during consensus generation). I don't think it's an issue in Flye since it crashes randomly at different stages. It likely has something to do with a particular machine or OS configuration. For some reason OS may be killing Flye processes - you make have overly strict limitations on threads or memory usage. But I don't have any additional advice here unfortunately.