Problem uploading samples

[Zymergen-SBRUBAKER]

Hi I have an error when uploading samples through the browser. There does not appear to be any documentation of how to do a manual upload using scp.

[glebkuznetsov]

Hi,

Can you describe the error? A common issue is not providing the correct full path to the files in the upload template. The full path to the location that each fastq has been scp'ed needs to be provided. Can you provide the upload template sheet you are using showing paths? (Feel free to provide a partially anonymized screenshot if you prefer).

Thanks,
Gleb

[Zymergen-SBRUBAKER]

Thank you! Here is the screenshot. I am trying to upload a pair of files through the web interface. Doing New, Upload Through Browser... I cannot really see a full path represented through this method and there is not a sample sheet. Maybe I should try the batch upload? I could also try scp but I don't have the full instructions on how to do it and not sure where the files go. Thanks for your help!

…

On Mon, Nov 4, 2019 at 9:48 AM Gleb Kuznetsov ***@***.***> wrote: Hi, Can you describe the error? A common issue is not providing the correct full path to the files in the upload template. The full path to the location that each fastq has been scp'ed needs to be provided. Can you provide the upload template sheet you are using showing paths? (Feel free to provide a partially anonymized screenshot if you prefer). Thanks, Gleb — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44UMNBCYMM6XTS63OU3QSBOANA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAD2AA#issuecomment-549469440>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44SJXRN4IE4NF5M26S3QSBOANANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Hello,

Looks like the attachment didn't make it through. Might have to use the Github Issues interface directly for it to go through (rather than email?) If you have any additional details about the failure, please let me know (e.g. is there a delay before the error?)

Sorry we never put up documentation for scp upload. In fact it is the method most users here in the Church Lab use today. Briefly, the steps are something like:

1. scp files to a location of your choice on the machine running Millstone. E.g., create a data directory /home/ubuntu/raw-data and scp there.

2. Through the browser Samples page, use New -> From Server Location...

3. Fill in the template linked in that form with each row representing a sample and the full path to the location.

Unfortunately, we didn't get around to adding scp instruction to the official docs here:
https://millstone.readthedocs.io/en/latest/user_guide/projects_alignments.html

In case it's helpful, here's a draft of more complete documentation we started writing but never got around to posting. Feel free to glance in case there is something helpful there.
https://docs.google.com/document/d/1tbPiVaaVqECliw5Eu8xBJ8OxpVHynWpo1_kFEFkoJmU/edit?usp=sharing

Thanks!
Gleb

[Zymergen-SBRUBAKER]

Hi Gleb! I just wanted to give you an update that I may have made a little more progress. I tried using the second option, batch upload from a template through browser, and it seemed to get through the first file, but got hung up on the second every time. I then tried the scp. It had similar issue, where file 1 says copying and file 2 stays in a state of queued to copy. So I tried a couple of other different files using the batch upload - it seems to have gotten both files. Now I'm running into some errors on fastqc and alignment step - let me try a couple more things and then I'll let you know how it goes. Thanks for your help! - Shane

…

On Mon, Nov 4, 2019 at 9:48 AM Gleb Kuznetsov ***@***.***> wrote: Hi, Can you describe the error? A common issue is not providing the correct full path to the files in the upload template. The full path to the location that each fastq has been scp'ed needs to be provided. Can you provide the upload template sheet you are using showing paths? (Feel free to provide a partially anonymized screenshot if you prefer). Thanks, Gleb — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44UMNBCYMM6XTS63OU3QSBOANA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAD2AA#issuecomment-549469440>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44SJXRN4IE4NF5M26S3QSBOANANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Hi Shane,

The "queued to copy" state is actually temporary and should resolve with some time. Though usually it's pretty quick.

A couple other issues to check:

• what is the ec2 instnce you are using?
• how much EBS space is there?

Thanks,
Gleb

[Zymergen-SBRUBAKER]

Thanks Gleb! It looks like it was stuck in queueing. I believe there may have been something weird about those files. On the other files, they loaded, and then I had to make sure they were really gzipped files, and then I got the fastqc step to work! That is a good point, I am running on the smallest ec2 instance right now. It does say there is about 1.7GB still free in the upper right corner. I'll keep an eye on that and I will try the alignment step next. Thanks for all your help so far, I really like Millstone! I will let you know how it goes. :)

…

On Tue, Nov 5, 2019 at 3:07 PM Gleb Kuznetsov ***@***.***> wrote: Hi Shane, The "queued to copy" state is actually temporary and should resolve with some time. Though usually it's pretty quick. A couple other issues to check: - what is the ec2 instnce you are using? - how much EBS space is there? Thanks, Gleb — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44U7YS6UI5BFGYDY7KTQSH4BJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEVA3Q#issuecomment-550064238>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44WALWBUGFQUUG5ASXDQSH4BJANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Hey Shane,

Good to hear it's working. Sorry the UI affordances are not perfect. We're expert users here so we hack features as needed :)

As far as instance type, I'd recommend running at least m5.xlarge to make sure you have enough memory for the alignment and subsequent analysis. I'm also concerned you may run out of disk space and would recommend allocating at least 3-5x as much disk as the size of all your FASTQs.

There's a few other tricks we do here to speed things up and make efficient use of AWS. But some of them require manually modifying some of the code and config files and we haven't really documented this anywhere. For example, when aligning > 50 genomes, we'll normally change the instance type to one with many cores (e.g. c5.9xlarge) and tweak the respective config that says to use all the cores. This actually turns out to be cheaper than running a smalerl instance in a more serial model. We'll then change back to a smaller instance type to do analysis / export data.

Anyway, happy to discuss/advise further if you're interested in ramping up your microbial genome alignment/analysis pipelines.

Cheers,
Gleb

[Zymergen-SBRUBAKER]

Thanks so much Gleb! I have now gotten data in there and it runs FastQC successfully. I then tried to do an alignment, but it seems to find no variants in the table. When I go to the alignment it says no data in table. The job status says completed. I don't know if it's possible there are no variants - do you have a good test dataset that you would recommend? Or perhaps this is related to the small machine size, I could try that. The job says it completed in 2 minutes which sounds short (this is e coli). The log seems to go to the end, but does mention something about an output being truncated. I'm pasting the log output below. Let me know what you think! ==START OF ALIGNMENT PIPELINE FOR SAB2, (411a5775) == /home/ubuntu/millstone/genome_designer/conf/../tools/bwa/bwa mem -t 1 -R "@rg\tID:77e5a0a4\tPL:illumina\tPU:77e5a0a4\tLB:77e5a0a4\tSM:77e5a0a4" -a /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/ref_genomes/d89f8976/tmpxft5Bs_MG1655_fasta <(gzip -dc /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/samples/77e5a0a4/s1.fastq.gz) <(gzip -dc /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/samples/77e5a0a4/s2.fastq.gz) | /home/ubuntu/millstone/genome_designer/conf/../tools/samtools/samtools view -bS -[M::main_mem] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48443, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (499.13, 49.98) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::worker2@0] performed mate-SW for 3064 reads [samopen] SAM header is present: 1 sequences. [M::main_mem] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48360, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.82, 49.62) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::worker2@0] performed mate-SW for 3100 reads [M::main_mem] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48333, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (499.12, 49.84) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::worker2@0] performed mate-SW for 3324 reads [M::main_mem] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48406, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.96, 49.80) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::worker2@0] performed mate-SW for 3006 reads [M::main_mem] read 60000 sequences (6000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 29029, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.83, 49.63) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::worker2@0] performed mate-SW for 1808 reads [main] Version: 0.7.5a-r405 [main] CMD: /home/ubuntu/millstone/genome_designer/conf/../tools/bwa/bwa mem -t 1 -R @rg\tID:77e5a0a4\tPL:illumina\tPU:77e5a0a4\tLB:77e5a0a4\tSM:77e5a0a4 -a /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/ref_genomes/d89f8976/tmpxft5Bs_MG1655_fasta /dev/fd/63 /dev/fd/62 [main] Real time: 19.494 sec; CPU: 14.312 sec [bam_header_read] EOF marker is absent. The input is probably truncated. [bam_rmdup_core] processing reference U00096... [bam_rmdup_core] 43 / 230000 = 0.0002 in library '77e5a0a4' Removed 0 outliers with isize >= 849 ==END OF ALIGNMENT PIPELINE==

…

On Tue, Nov 5, 2019 at 6:54 PM Gleb Kuznetsov ***@***.***> wrote: Hey Shane, Good to hear it's working. Sorry the UI affordances are not perfect. We're expert users here so we hack features as needed :) As far as instance type, I'd recommend running at least m5.xlarge to make sure you have enough memory for the alignment and subsequent analysis. I'm also concerned you may run out of disk space and would recommend allocating at least 3-5x as much disk as the size of all your FASTQs. There's a few other tricks we do here to speed things up and make efficient use of AWS. But some of them require manually modifying some of the code and config files and we haven't really documented this anywhere. For example, when aligning > 50 genomes, we'll normally change the instance type to one with many cores (e.g. c5.9xlarge) and tweak the respective config that says to use all the cores. This actually turns out to be cheaper than running a smalerl instance in a more serial model. We'll then change back to a smaller instance type to do analysis / export data. Anyway, happy to discuss/advise further if you're interested in ramping up your microbial genome alignment/analysis pipelines. Cheers, Gleb — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44T4225J5QKVTLOXVH3QSIWV3A5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDFCM7I#issuecomment-550119037>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44QPMELTCXLHQXY54W3QSIWV3ANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Hey Shane,

No variants sounds surprising. And I agree 2 min sounds fast for an alignment and variant calling. It's hard to tell from the logs whether anything specifically went wrong. I can't remember whether "truncated input" is bad. It is possible the small machine ran out of memory at some point and a process failed in a way that didn't disrupt the rest of the pipeline.

You can look at our unit tests for some example data. For example if you trace through this test https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138, you'll see variables pointing to an example genome and fastqs:

        self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data',
                'test_genome_known_substitutions')

        self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions.fa')

        self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions_0.snps.simLibrary.1.fq')

        self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions_0.snps.simLibrary.2.fq')

And test_data is located here: https://github.com/churchlab/millstone/tree/master/genome_designer/test_data

[Zymergen-SBRUBAKER]

Awesome, thank you Gleb! I also looked into the Jbrowse and I saw some coverage plots and other things, so I think it definitely did something. I will try your test data next. :)

…

On Wed, Nov 6, 2019 at 2:23 PM Gleb Kuznetsov ***@***.***> wrote: Hey Shane, No variants sounds surprising. And I agree 2 min sounds fast for an alignment and variant calling. It's hard to tell from the logs whether anything specifically went wrong. I can't remember whether "truncated input" is bad. It is possible the small machine ran out of memory at some point and a process failed in a way that didn't disrupt the rest of the pipeline. You can look at our unit tests for some example data. For example if you trace through this test https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138, you'll see variables pointing to an example genome and fastqs: self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data', 'test_genome_known_substitutions') self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, 'test_genome_known_substitutions.fa') self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, 'test_genome_known_substitutions_0.snps.simLibrary.1.fq') self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, 'test_genome_known_substitutions_0.snps.simLibrary.2.fq') And test_data is located here: https://github.com/churchlab/millstone/tree/master/genome_designer/test_data — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44XJ47LUGHEM5AZF6STQSM7VJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIGJBQ#issuecomment-550528134>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44QREKJHNMDEDD7TXBDQSM7VJANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[Zymergen-SBRUBAKER]

Hi Gleb! I tried out the sample data from E coli that you pointed me to and it looks like it mostly worked. Thanks! I can see SNPs in the tracks in the browser. However, when I click on the Variants tab on the left side, it always just says "No Data in Table". Also, when click on the alignment and "go to variants" I get the error message in the screenshot attached. Thanks again for your help. - Shane On Wed, Nov 6, 2019 at 3:43 PM Shane Brubaker <sbrubaker@zymergen.com> wrote:

…

Awesome, thank you Gleb! I also looked into the Jbrowse and I saw some coverage plots and other things, so I think it definitely did something. I will try your test data next. :) On Wed, Nov 6, 2019 at 2:23 PM Gleb Kuznetsov ***@***.***> wrote: > Hey Shane, > > No variants sounds surprising. And I agree 2 min sounds fast for an > alignment and variant calling. It's hard to tell from the logs whether > anything specifically went wrong. I can't remember whether "truncated > input" is bad. It is possible the small machine ran out of memory at some > point and a process failed in a way that didn't disrupt the rest of the > pipeline. > > You can look at our unit tests for some example data. For example if you > trace through this test > https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138, > you'll see variables pointing to an example genome and fastqs: > > self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data', > 'test_genome_known_substitutions') > > self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, > 'test_genome_known_substitutions.fa') > > self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, > 'test_genome_known_substitutions_0.snps.simLibrary.1.fq') > > self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT, > 'test_genome_known_substitutions_0.snps.simLibrary.2.fq') > > And test_data is located here: > https://github.com/churchlab/millstone/tree/master/genome_designer/test_data > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#701?email_source=notifications&email_token=AM2Y44XJ47LUGHEM5AZF6STQSM7VJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIGJBQ#issuecomment-550528134>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AM2Y44QREKJHNMDEDD7TXBDQSM7VJANCNFSM4JHTQX6Q> > . > -- Shane Brubaker Computational Biology Architect Zymergen, Inc, ***@***.***

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Hey Shane,

Looks like the attachment didn't make it through to Github. Maybe it's not going through by email? Can you try uploading directly at the issue URL: #701?

Thanks,
Gleb

[Zymergen-SBRUBAKER]

Here is the screenshot!

[Zymergen-SBRUBAKER]

Thank you Gleb! I have attached the screenshot in the issue. :)

…

On Tue, Nov 19, 2019 at 8:42 PM Gleb Kuznetsov ***@***.***> wrote: Hey Shane, Looks like the attachment didn't make it through to Github. Maybe it's not going through by email? Can you try uploading directly at the issue URL: #701 <#701>? Thanks, Gleb — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#701?email_source=notifications&email_token=AM2Y44XBBQ6CXEPYU3H6TMLQUS54LA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEQV5LY#issuecomment-555835055>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2Y44XDXWF7EHI6XRFCXELQUS54LANCNFSM4JHTQX6Q> .

-- Shane Brubaker Computational Biology Architect Zymergen, Inc, sbrubaker@zymergen.com

[glebkuznetsov]

Ah, interesting. It looks like it might be version issue either with Django or Postgres. Though might be the data.

Are you running Millstone on AWS using our pre-built AMI?