SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
problems with downloading files using fasp protocol stemcell Bioinformatics 2 10-15-2012 12:39 PM
Problems with Bam files from Tophat bjchen Bioinformatics 0 04-11-2012 12:22 PM
password to decrypt files in dbgap rafi bondi Bioinformatics 2 07-21-2011 06:51 AM
TCGA dbgap: where's the bams? Richard Finney Bioinformatics 0 06-13-2011 08:18 AM
problems visualizing bam files in IGV Line RNA Sequencing 2 11-29-2010 05:33 AM

Reply
 
Thread Tools
Old 04-19-2013, 10:27 PM   #1
caswater
Member
 
Location: Canada

Join Date: Jun 2011
Posts: 47
Default help~ problems when decrypting dbGAP files

does anyone have used dbGAP data before? We downloaded a file from dbGAP which needs decryption tool from NCBI to open the data. But it was really a pain to decrypt the data, and I can't figure it out after playing with it for over 4 hours..... anyone can give me some guidance?

thanks!!

So when I run the command in the folder "SEQ_DATA", ## where the data directory and the tool directory are located:

./decryption.2.3.2-4/bin/configuration-assistant.perl

The output is below.
###############
cwd = '/Users/UserA/Desktop/SEQ_DATA'

checking for fastq-dump (local build)... no
checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump: not found)
checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump.exe: not found)
checking for fastq-dump (fastq-dump)... no
checking for sam-dump (local build)... no
checking for sam-dump (decryption.2.3.2-4/bin/sam-dump: not found)
checking for sam-dump (decryption.2.3.2-4/bin/sam-dump.exe: not found)
checking for sam-dump (sam-dump)... no
presuming to be run in decryption package
checking for vdb-config (local build)... no
checking for vdb-config (decryption.2.3.2-4/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'false'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/Users/Jingjing/ncbi/public': does not exist
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct
###################

The software even didn't ask for encryption password from us. Then when I run vdb-decrypt, I got the following msgs...

./decryption.2.3.2-4/bin/vdb-decrypt 12345(#project ID#)
2013-04-20T05:16:03 vdb-decrypt.2.3.2 err: file not found while opening encryption key within virtual file system module - unable to obtain a password
2013-04-20T05:16:03 vdb-decrypt.2.3.2: exiting: RC(rcVFS,rcEncryptionKey,rcOpening,rcFile,rcNotFound) (2665808984)

Do you know if we have still missed anything from dbGAP to decrypt the files?
Your kind help is highly appreciated.
caswater is offline   Reply With Quote
Old 04-20-2013, 07:55 PM   #2
frewise
Member
 
Location: China

Join Date: Jun 2011
Posts: 13
Default

You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt

Last edited by frewise; 04-20-2013 at 07:58 PM.
frewise is offline   Reply With Quote
Old 04-20-2013, 10:09 PM   #3
caswater
Member
 
Location: Canada

Join Date: Jun 2011
Posts: 47
Default

thank you, buddy.. It really worked. I was using mac, but can't get it done, but the linux version is perfect.

many thx~
caswater is offline   Reply With Quote
Old 05-08-2013, 12:59 PM   #4
Qwerty0
Junior Member
 
Location: PSU

Join Date: Oct 2011
Posts: 2
Default

frewise: Could you clarify which password you're referring to?
I count at least 3 different possible "passwords":
1. The NCBI login password (e.g. to eRA Commons)
2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
3. The "dbGaP repository key" file in the same section

I'm having the same exact problem, but writing each of those hasn't worked.
My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
Then I ran vdb-decrypt on the .ncbi_enc file.
I got the same error caswater did.

Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):
Code:
cwd = '/home/me/bx/data/dbgap'

checking for fastq-dump (local build)... no
checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
checking for sam-dump (local build)... no
checking for sam-dump (/home/me/bin/sam-dump: found)... yes
checking for vdb-config (local build)... no
checking for vdb-config (/home/me/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'no'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/home/me/ncbi/public': exists
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct
Qwerty0 is offline   Reply With Quote
Old 05-11-2013, 10:10 PM   #5
frewise
Member
 
Location: China

Join Date: Jun 2011
Posts: 13
Default

Quote:
Originally Posted by Qwerty0 View Post
frewise: Could you clarify which password you're referring to?
I count at least 3 different possible "passwords":
1. The NCBI login password (e.g. to eRA Commons)
2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
3. The "dbGaP repository key" file in the same section

I'm having the same exact problem, but writing each of those hasn't worked.
My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
Then I ran vdb-decrypt on the .ncbi_enc file.
I got the same error caswater did.

Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):
Code:
cwd = '/home/me/bx/data/dbgap'

checking for fastq-dump (local build)... no
checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
checking for sam-dump (local build)... no
checking for sam-dump (/home/me/bin/sam-dump: found)... yes
checking for vdb-config (local build)... no
checking for vdb-config (/home/me/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'no'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/home/me/ncbi/public': exists
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct
The passwd should be the one you set to the specific dataset, but not the eRA Commons. If you forgot the passwd, maybe you could reset it by running vdb-passwd, I am not sure if it works. Please leave a msg if it works, thanks!
frewise is offline   Reply With Quote
Old 05-12-2013, 04:01 PM   #6
genomeHunter
Member
 
Location: Canada

Join Date: Apr 2013
Posts: 26
Default Sra

Quote:
Originally Posted by frewise View Post
You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt
Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH
genomeHunter is offline   Reply With Quote
Old 05-13-2013, 01:12 AM   #7
frewise
Member
 
Location: China

Join Date: Jun 2011
Posts: 13
Default

Quote:
Originally Posted by genomeHunter View Post
Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH
I am not very sure about it. What I can tell are:
1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

So, I guess NCBI Archive files are some special files that not always emerge.
frewise is offline   Reply With Quote
Old 05-13-2013, 03:59 AM   #8
genomeHunter
Member
 
Location: Canada

Join Date: Apr 2013
Posts: 26
Default

Quote:
Originally Posted by frewise View Post
I am not very sure about it. What I can tell are:
1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

So, I guess NCBI Archive files are some special files that not always emerge.
NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

GH
genomeHunter is offline   Reply With Quote
Old 05-13-2013, 07:20 PM   #9
frewise
Member
 
Location: China

Join Date: Jun 2011
Posts: 13
Default

Quote:
Originally Posted by genomeHunter View Post
NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

GH
I downloaded sra files, but the file names are also ended with "_enc". They also need decryption.
frewise is offline   Reply With Quote
Old 05-15-2013, 06:56 PM   #10
nstoler
Junior Member
 
Location: PSU

Join Date: May 2013
Posts: 1
Default

Quote:
Originally Posted by genomeHunter View Post
Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH
Oh, I ran into this too. That happens when you try to decrypt .sra files. You can still decrypt by just using the option --decrypt-sra-files.

What they're saying, though, is that this isn't the recommended way to deal with these files. The tools from the SRA Toolkit like fastq-dump are supposedly able to work directly on encrypted files, so you don't technically have to decrypt them first. It's a safer way to deal with these files, keeping them encrypted, since they contain confidential data. But on the other hand it's kind of an absurd policy, since all downstream analysis results in unencrypted files containing all the original data anyway. FASTQ and BAM files aren't encrypted but contain all the reads in the original, encrypted .sra file.

The proper way to do this would be to do all the work inside a filesystem which itself is encrypted, using TrueCrypt, EncFS, or even full disk encryption. But for whatever reason, NCBI chose to build their own, very different system.
nstoler is offline   Reply With Quote
Old 05-15-2013, 07:12 PM   #11
genomeHunter
Member
 
Location: Canada

Join Date: Apr 2013
Posts: 26
Default

Thank you so much! Your answer was enlightening. I will probably decrypt them since they are located on a secure server.

Cheers,
GH
genomeHunter is offline   Reply With Quote
Old 10-17-2013, 11:50 AM   #12
blank964
Junior Member
 
Location: chicago

Join Date: Oct 2013
Posts: 1
Default

Quote:
Originally Posted by frewise View Post
You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt
Thanks this worked for me.
blank964 is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 08:33 PM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.