Fetch fastqs from basespace with command line

[julio514]

Dear seqanswers community,
I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

Many thanks,
Julio

[lh3]

See https://gist.github.com/lh3/54f535b11a9ee5d3be8e

[julio514]

Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
Cheers,

[lh3]

I don't know how, but there must be ways with their APIs.

[dariober]

Dear seqanswers community,
I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

Many thanks,
Julio

I've used BaseSpaceR to get fastq files via R. (This was while ago).

If I correctly remember and things haven't changed, it's a bit long winded to get started as you need to get a token. Also it's not enough to have a run shared with you, the owner of the project has to share the entire project (I think...). Then something on these lines should work:

library(BaseSpaceR)
ACCESS_TOKEN<- 'dd9...mytoken...43'
PROJECT_ID<- '123456'  ## Get proj ID from url of the project

aAuth<- AppAuth(access_token = ACCESS_TOKEN)
selProj <- Projects(aAuth, id = PROJECT_ID, simplify = TRUE) 
sampl <- listSamples(selProj, limit= 1000)
inSample <- Samples(aAuth, id = Id(sampl), simplify = TRUE)
for(s in inSample){ 
    f <- listFiles(s, Extensions = ".gz")
    print(Name(f))
    getFiles(aAuth, id= Id(f), destDir = 'outdir/', verbose = TRUE)
}

[julio514]

Thanks dariober, works like a charm!

[jdv]

A while ago I wrote an interactive BaseSpace command-line client for downloading data to headless servers. You might wish to try it as an alternative to the other options here. It has an FTP-like interface that allows you to browse the data in your account based on the Project/Sample/File hierarchy currently used. You can download individual files or entire projects at a time. As with the other alternatives, you need to obtain a developer's access token to use with it, but it can store this token to disk using symmetric encryption to make future use a bit easier. It's meant for interactive use, so if you want batch or scripting capabilities use one of the other suggestions.

I just got around today to uploading it to SourceForge:

http://sourceforge.net/projects/bsfetch/

It is written in Perl and I use it on Linux. I've also tested it briefly on Windows, where it seems to work with the exception of password masking on the command line.

[jdv]

Also, a quick note for anyone using lh3's method to manually construct URLs - not only does this open up your access_token for viewing in the process table by anyone else on the same machine (e.g. with top, ps, etc) as already mentioned, but it is also transmitted in cleartext with the HTTPS request URL and can be trivially captured by anyone happening to be watching the network traffic.

The secure way of sending the request is with the access token specified in the 'x-access-token' HTTP header, which is encrypted in the SSL connection. See here:

https://developer.basespace.illumina.com/docs/content/documentation/rest-api/api-reference#Using_the_Access_Token

A modification of lh3's method using curl instead of wget would be like this:

curl -L -J --config token_header.txt https://api.basespace.illumina.com/v1pre3/files/YOUR-FILE-ID/content -O

where the arbitrarily named 'token_header.txt' contains something like this:

header = "x-access-token: YOUR-TOKEN-HERE"

This prevents snooping via the process table or network traffic. Obviously you won't want to leave the 'token_header.txt' file sitting around on shared disk space.

[lh3]

Nice tips. Thanks!

[SF_mallish]

Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
Cheers,

It might be too late, but in the first link lh provided in his gist, there is a python script to allow you to download all files in one run by specifying the run ID.

https://support.basespace.illumina.com/knowledgebase/articles/403618-python-run-downloader

I also have a small python script to allow you download all fastq files in one project by specifying project name and access Token.

https://github.com/yu68/tools/tree/master/BaseSpace-tools

[julio514]

Also these python scripts that work well (tested)

https://github.com/nh13/basespace-invaders

Cheers,

[seahym]

Too late to answer OP, but in case anyone else needs an alternative, the BaseSpace command line interface was really straightforward to install and use on Ubuntu. To download fastq files by project:

bs download project --id <project_id> -o <target_directory>