SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
SRA to fastq conversion with fastq-dump loses sequences pcantalupo Bioinformatics 13 10-08-2015 04:09 PM
problem with sra toolkit fastq-dump sratoolkit.2.1.10-win64 hui_shi Bioinformatics 13 05-21-2015 05:21 PM
fastq-dump transforu Bioinformatics 5 12-11-2014 09:22 AM
Convert SRA to FASTQ with fastq-dump but problem of read length MurielGB Illumina/Solexa 9 10-07-2014 09:10 AM
Fastq-dump.exe Giles Bioinformatics 2 06-11-2011 12:34 PM

Reply
 
Thread Tools
Old 08-09-2017, 03:17 PM   #1
vsmuir
Junior Member
 
Location: Seattle, WA

Join Date: Oct 2016
Posts: 3
Default SRAtoolkit - prefetch and fastq-dump problems

I'm attempting to download a ton of protected GTEx data using sratoolkit.

Using fastq-dump alone for accession numbers is painfully slow. My Mac is taking 2-4 hours to get each ~3Gb file.

e.g. This took 3.4GB file took 2:23:19 to download:
Code:
~/sra-toolkit/bin/fastq-dump --bzip2 --split-spot --clip --skip-technical --dumpbase --readids SRR1397673
Prefetch has worked better; running both the html download (~30min) and fastq-dump from the prefetched .sra (~1hr) is much faster than fastq-dump alone. Unfortunately, I can't get prefetch to use Aspera.
(GTEx SRR#s have been changed to a publicly available SRR# for the examples below.)

I think the aspera problem is prefetch-specific; tests using the following code resulted in a nice, speedy download.
Code:
~/Applications/Aspera\ Connect.app/Contents/Resources/ascp -T -l150M -i ~/Applications/Aspera\ Connect.app/Contents/Resources/asperaweb_id_dsa.putty anonftp@ftp-trace.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR292/SRR292241/SRR292241.sra .
However, things went much less smoothly using this:
Code:
~/sra-toolkit/bin/prefetch -t ascp -a "~/Applications/Aspera\ Connect.app/Contents/Resources/ascp | ~/Applications/Aspera\ Connect.app/Contents/Resources/asperaweb_id_dsa.openssh" SRR292241
I ended up with this error:
2017-08-09T22:54:52 prefetch.2.8.2: 1) Downloading 'SRR292241'...
2017-08-09T22:54:52 prefetch.2.8.2: Downloading via fasp...
2017-08-09T22:54:53 prefetch.2.8.2 err: process failed while waiting process - ascp failed with 1
2017-08-09T22:54:54 prefetch.2.8.2 err: process failed while waiting process - ascp failed with 1
2017-08-09T22:54:54 prefetch.2.8.2: fasp download failed
2017-08-09T22:54:54 prefetch.2.8.2: 1) failed to download SRR292241

I've tried adding the -X 200G flag to circumvent the error, without success.

Any suggestions for speeding up my download from SRA/dbGaP?
vsmuir is offline   Reply With Quote
Old 08-14-2017, 11:07 AM   #2
vsmuir
Junior Member
 
Location: Seattle, WA

Join Date: Oct 2016
Posts: 3
Default Some solutions

If anyone else has run into this problem, I've found a couple of things that seem to help.

1) If you have similar problems with prefetch or fasp downloads fail:
Simply remove the -a and -t flags. When I dropped the -t flag alone, prefetch would fail to download via fasp and default to html. When I also removed the -a flag, prefetch successfully downloaded via fasp.

2) If running fastq-dump runs painfully slow (before or after prefetching):
I strongly recommend using parallel-fastq-dump.
vsmuir is offline   Reply With Quote
Old 08-14-2017, 12:15 PM   #3
Markiyan
Member
 
Location: Cambridge

Join Date: Sep 2010
Posts: 88
Lightbulb

If you want predictable fastq-dump performance... use local data sources.

1. Forget about the wi-fi. Use only wired network (suitable docking station with ethernet if on macbook).

2. Download the sra files to the DAS or good NAS (use aspera, do it over the weekend if possible and be patient).

3. Than mount in on the mac/linux box and give SRA files to the fastq-dump actual SRA files.
Markiyan is offline   Reply With Quote
Reply

Tags
fastq-dump, gtex, prefetch, sratoolkit

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 12:58 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO