help~ problems when decrypting dbGAP files

[caswater]

does anyone have used dbGAP data before? We downloaded a file from dbGAP which needs decryption tool from NCBI to open the data. But it was really a pain to decrypt the data, and I can't figure it out after playing with it for over 4 hours..... anyone can give me some guidance?

thanks!!

So when I run the command in the folder "SEQ_DATA", ## where the data directory and the tool directory are located:

./decryption.2.3.2-4/bin/configuration-assistant.perl

The output is below.
###############
cwd = '/Users/UserA/Desktop/SEQ_DATA'

checking for fastq-dump (local build)... no
checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump: not found)
checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump.exe: not found)
checking for fastq-dump (fastq-dump)... no
checking for sam-dump (local build)... no
checking for sam-dump (decryption.2.3.2-4/bin/sam-dump: not found)
checking for sam-dump (decryption.2.3.2-4/bin/sam-dump.exe: not found)
checking for sam-dump (sam-dump)... no
presuming to be run in decryption package
checking for vdb-config (local build)... no
checking for vdb-config (decryption.2.3.2-4/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'false'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/Users/Jingjing/ncbi/public': does not exist
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct
###################

The software even didn't ask for encryption password from us. Then when I run vdb-decrypt, I got the following msgs...

./decryption.2.3.2-4/bin/vdb-decrypt 12345(#project ID#)
2013-04-20T05:16:03 vdb-decrypt.2.3.2 err: file not found while opening encryption key within virtual file system module - unable to obtain a password
2013-04-20T05:16:03 vdb-decrypt.2.3.2: exiting: RC(rcVFS,rcEncryptionKey,rcOpening,rcFile,rcNotFound) (2665808984)

Do you know if we have still missed anything from dbGAP to decrypt the files?
Your kind help is highly appreciated.

[frewise]

You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt

[caswater]

thank you, buddy.. It really worked. I was using mac, but can't get it done, but the linux version is perfect.

many thx~

[Qwerty0]

frewise: Could you clarify which password you're referring to?
I count at least 3 different possible "passwords":
1. The NCBI login password (e.g. to eRA Commons)
2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
3. The "dbGaP repository key" file in the same section

I'm having the same exact problem, but writing each of those hasn't worked.
My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
Then I ran vdb-decrypt on the .ncbi_enc file.
I got the same error caswater did.

Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):

cwd = '/home/me/bx/data/dbgap'

checking for fastq-dump (local build)... no
checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
checking for sam-dump (local build)... no
checking for sam-dump (/home/me/bin/sam-dump: found)... yes
checking for vdb-config (local build)... no
checking for vdb-config (/home/me/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'no'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/home/me/ncbi/public': exists
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct

[frewise]

frewise: Could you clarify which password you're referring to?
I count at least 3 different possible "passwords":
1. The NCBI login password (e.g. to eRA Commons)
2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
3. The "dbGaP repository key" file in the same section

I'm having the same exact problem, but writing each of those hasn't worked.
My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
Then I ran vdb-decrypt on the .ncbi_enc file.
I got the same error caswater did.

Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):

cwd = '/home/me/bx/data/dbgap'

checking for fastq-dump (local build)... no
checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
checking for sam-dump (local build)... no
checking for sam-dump (/home/me/bin/sam-dump: found)... yes
checking for vdb-config (local build)... no
checking for vdb-config (/home/me/bin/vdb-config: found)... yes

Reading configuration
refseq/servers: not found
refseq/volumes: not found
refseq/paths: not found

repository: found
repository/site: not found

repository/remote: found
repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
repository/remote/main/NCBI/disabled: 'no'
repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
repository/user: found
repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
repository/user/main/public/root: '/home/me/ncbi/public': exists
repository/user/main/public/cache-enabled: 'true'
repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

Configuration is correct

The passwd should be the one you set to the specific dataset, but not the eRA Commons. If you forgot the passwd, maybe you could reset it by running vdb-passwd, I am not sure if it works. Please leave a msg if it works, thanks!

[genomeHunter]

Sra

You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt

Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH

[frewise]

Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH

I am not very sure about it. What I can tell are:
1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

So, I guess NCBI Archive files are some special files that not always emerge.

[genomeHunter]

I am not very sure about it. What I can tell are:
1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

So, I guess NCBI Archive files are some special files that not always emerge.

NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

GH

[frewise]

NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

GH

I downloaded sra files, but the file names are also ended with "_enc". They also need decryption.

[nstoler]

Hello,

Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

"NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

I am confused!

Thanks,
GH

Oh, I ran into this too. That happens when you try to decrypt .sra files. You can still decrypt by just using the option --decrypt-sra-files.

What they're saying, though, is that this isn't the recommended way to deal with these files. The tools from the SRA Toolkit like fastq-dump are supposedly able to work directly on encrypted files, so you don't technically have to decrypt them first. It's a safer way to deal with these files, keeping them encrypted, since they contain confidential data. But on the other hand it's kind of an absurd policy, since all downstream analysis results in unencrypted files containing all the original data anyway. FASTQ and BAM files aren't encrypted but contain all the reads in the original, encrypted .sra file.

The proper way to do this would be to do all the work inside a filesystem which itself is encrypted, using TrueCrypt, EncFS, or even full disk encryption. But for whatever reason, NCBI chose to build their own, very different system.

[genomeHunter]

Thank you so much! Your answer was enlightening. I will probably decrypt them since they are located on a secure server.

Cheers,
GH

[blank964]

You should have set a password when applied the dbGaP.
Then,
1. make a new folder at your home directory: /home/xxx/.ncbi
2. create a file named vdb-passwd and save your passwd in it
3. run vdb-decrypt
This works for me.

A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt

Thanks this worked for me.

[reneesummer]

May I ask what this dbGaP password should be? Is it the password used to login to dbGap account where we download date?

Thanks this worked for me.