Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • help~ problems when decrypting dbGAP files

    does anyone have used dbGAP data before? We downloaded a file from dbGAP which needs decryption tool from NCBI to open the data. But it was really a pain to decrypt the data, and I can't figure it out after playing with it for over 4 hours..... anyone can give me some guidance?

    thanks!!

    So when I run the command in the folder "SEQ_DATA", ## where the data directory and the tool directory are located:

    ./decryption.2.3.2-4/bin/configuration-assistant.perl

    The output is below.
    ###############
    cwd = '/Users/UserA/Desktop/SEQ_DATA'

    checking for fastq-dump (local build)... no
    checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump: not found)
    checking for fastq-dump (decryption.2.3.2-4/bin/fastq-dump.exe: not found)
    checking for fastq-dump (fastq-dump)... no
    checking for sam-dump (local build)... no
    checking for sam-dump (decryption.2.3.2-4/bin/sam-dump: not found)
    checking for sam-dump (decryption.2.3.2-4/bin/sam-dump.exe: not found)
    checking for sam-dump (sam-dump)... no
    presuming to be run in decryption package
    checking for vdb-config (local build)... no
    checking for vdb-config (decryption.2.3.2-4/bin/vdb-config: found)... yes

    Reading configuration
    refseq/servers: not found
    refseq/volumes: not found
    refseq/paths: not found

    repository: found
    repository/site: not found

    repository/remote: found
    repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
    repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
    repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
    repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
    repository/remote/main/NCBI/disabled: 'false'
    repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
    repository/user: found
    repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
    repository/user/main/public/root: '/Users/Jingjing/ncbi/public': does not exist
    repository/user/main/public/cache-enabled: 'true'
    repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
    repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'

    Configuration is correct
    ###################

    The software even didn't ask for encryption password from us. Then when I run vdb-decrypt, I got the following msgs...

    ./decryption.2.3.2-4/bin/vdb-decrypt 12345(#project ID#)
    2013-04-20T05:16:03 vdb-decrypt.2.3.2 err: file not found while opening encryption key within virtual file system module - unable to obtain a password
    2013-04-20T05:16:03 vdb-decrypt.2.3.2: exiting: RC(rcVFS,rcEncryptionKey,rcOpening,rcFile,rcNotFound) (2665808984)

    Do you know if we have still missed anything from dbGAP to decrypt the files?
    Your kind help is highly appreciated.

  • #2
    You should have set a password when applied the dbGaP.
    Then,
    1. make a new folder at your home directory: /home/xxx/.ncbi
    2. create a file named vdb-passwd and save your passwd in it
    3. run vdb-decrypt
    This works for me.

    A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt
    Last edited by frewise; 04-20-2013, 07:58 PM.

    Comment


    • #3
      thank you, buddy.. It really worked. I was using mac, but can't get it done, but the linux version is perfect.

      many thx~

      Comment


      • #4
        frewise: Could you clarify which password you're referring to?
        I count at least 3 different possible "passwords":
        1. The NCBI login password (e.g. to eRA Commons)
        2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
        3. The "dbGaP repository key" file in the same section

        I'm having the same exact problem, but writing each of those hasn't worked.
        My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
        Then I ran vdb-decrypt on the .ncbi_enc file.
        I got the same error caswater did.

        Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):
        Code:
        cwd = '/home/me/bx/data/dbgap'
        
        checking for fastq-dump (local build)... no
        checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
        checking for sam-dump (local build)... no
        checking for sam-dump (/home/me/bin/sam-dump: found)... yes
        checking for vdb-config (local build)... no
        checking for vdb-config (/home/me/bin/vdb-config: found)... yes
        
        Reading configuration
        refseq/servers: not found
        refseq/volumes: not found
        refseq/paths: not found
        
        repository: found
        repository/site: not found
        
        repository/remote: found
        repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
        repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
        repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
        repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
        repository/remote/main/NCBI/disabled: 'no'
        repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
        repository/user: found
        repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
        repository/user/main/public/root: '/home/me/ncbi/public': exists
        repository/user/main/public/cache-enabled: 'true'
        repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
        repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'
        
        Configuration is correct

        Comment


        • #5
          Originally posted by Qwerty0 View Post
          frewise: Could you clarify which password you're referring to?
          I count at least 3 different possible "passwords":
          1. The NCBI login password (e.g. to eRA Commons)
          2. The password dbGaP sends you when you click the "Remind password" link next to each "Download request" in the "Downloads" section
          3. The "dbGaP repository key" file in the same section

          I'm having the same exact problem, but writing each of those hasn't worked.
          My repository path is ~/ncbi/public, and I tried it with both of the first two passwords, by using vi to create a file named 'vdb-passwd' containing just the password followed by a newline, and placing it in ~/.ncbi, ~/ncbi, and ~/ncbi/public, just in case.
          Then I ran vdb-decrypt on the .ncbi_enc file.
          I got the same error caswater did.

          Here is the output of configuration-assistant.perl for me (I'm on Ubuntu 12.10):
          Code:
          cwd = '/home/me/bx/data/dbgap'
          
          checking for fastq-dump (local build)... no
          checking for fastq-dump (/home/me/bin/fastq-dump: found)... yes
          checking for sam-dump (local build)... no
          checking for sam-dump (/home/me/bin/sam-dump: found)... yes
          checking for vdb-config (local build)... no
          checking for vdb-config (/home/me/bin/vdb-config: found)... yes
          
          Reading configuration
          refseq/servers: not found
          refseq/volumes: not found
          refseq/paths: not found
          
          repository: found
          repository/site: not found
          
          repository/remote: found
          repository/remote/protected/CGI/resolver-cgi: 'http://www.ncbi.nlm.nih.gov/Traces/names/names.cgi'
          repository/remote/main/NCBI/apps/refseq/volumes/refseq: 'refseq'
          repository/remote/main/NCBI/apps/wgs/volumes/fuseWGS: 'wgs'
          repository/remote/main/NCBI/apps/sra/volumes/fuse1000: 'sra-instant/reads/ByRun/sra'
          repository/remote/main/NCBI/disabled: 'no'
          repository/remote/main/NCBI/root: 'http://ftp-trace.ncbi.nlm.nih.gov/sra'
          repository/user: found
          repository/user/main/public/apps/refseq/volumes/refseq: 'refseq'
          repository/user/main/public/root: '/home/me/ncbi/public': exists
          repository/user/main/public/cache-enabled: 'true'
          repository/user/main/public/apps/sra/volumes/sraFlat: 'sra'
          repository/user/main/public/apps/wgs/volumes/wgsFlat: 'wgs'
          
          Configuration is correct
          The passwd should be the one you set to the specific dataset, but not the eRA Commons. If you forgot the passwd, maybe you could reset it by running vdb-passwd, I am not sure if it works. Please leave a msg if it works, thanks!

          Comment


          • #6
            Sra

            Originally posted by frewise View Post
            You should have set a password when applied the dbGaP.
            Then,
            1. make a new folder at your home directory: /home/xxx/.ncbi
            2. create a file named vdb-passwd and save your passwd in it
            3. run vdb-decrypt
            This works for me.

            A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt
            Hello,

            Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

            "NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

            I am confused!

            Thanks,
            GH

            Comment


            • #7
              Originally posted by genomeHunter View Post
              Hello,

              Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

              "NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

              I am confused!

              Thanks,
              GH
              I am not very sure about it. What I can tell are:
              1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
              2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

              So, I guess NCBI Archive files are some special files that not always emerge.

              Comment


              • #8
                Originally posted by frewise View Post
                I am not very sure about it. What I can tell are:
                1) Files I downloaded from dbGaP are all ended with "_enc" which couldnot be used until I decrypted them.
                2) I simplely run /path/tp/vdb-decrypt /dir/to/your/data/ to decrypt.

                So, I guess NCBI Archive files are some special files that not always emerge.
                NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

                GH

                Comment


                • #9
                  Originally posted by genomeHunter View Post
                  NCBI archive files are .sra files that are used to generate .fastq files. Did you use .sra files?

                  GH
                  I downloaded sra files, but the file names are also ended with "_enc". They also need decryption.

                  Comment


                  • #10
                    Originally posted by genomeHunter View Post
                    Hello,

                    Since you were successful in decrypting, could you please shed light on this comment from (SRA Toolkit):

                    "NCBI Archive files that contain NCBI database objects will not be decrypted unless the decrypt-sra-files option is used. As these objects can be used without decryption it is recommended they remain encrypted."

                    I am confused!

                    Thanks,
                    GH
                    Oh, I ran into this too. That happens when you try to decrypt .sra files. You can still decrypt by just using the option --decrypt-sra-files.

                    What they're saying, though, is that this isn't the recommended way to deal with these files. The tools from the SRA Toolkit like fastq-dump are supposedly able to work directly on encrypted files, so you don't technically have to decrypt them first. It's a safer way to deal with these files, keeping them encrypted, since they contain confidential data. But on the other hand it's kind of an absurd policy, since all downstream analysis results in unencrypted files containing all the original data anyway. FASTQ and BAM files aren't encrypted but contain all the reads in the original, encrypted .sra file.

                    The proper way to do this would be to do all the work inside a filesystem which itself is encrypted, using TrueCrypt, EncFS, or even full disk encryption. But for whatever reason, NCBI chose to build their own, very different system.

                    Comment


                    • #11
                      Thank you so much! Your answer was enlightening. I will probably decrypt them since they are located on a secure server.

                      Cheers,
                      GH

                      Comment


                      • #12
                        Originally posted by frewise View Post
                        You should have set a password when applied the dbGaP.
                        Then,
                        1. make a new folder at your home directory: /home/xxx/.ncbi
                        2. create a file named vdb-passwd and save your passwd in it
                        3. run vdb-decrypt
                        This works for me.

                        A link for your reference: http://www.ncbi.nlm.nih.gov/Traces/s...&f=vdb-decrypt
                        Thanks this worked for me.

                        Comment


                        • #13
                          May I ask what this dbGaP password should be? Is it the password used to login to dbGap account where we download date?
                          Originally posted by blank964 View Post
                          Thanks this worked for me.

                          Comment

                          Latest Articles

                          Collapse

                          • seqadmin
                            Strategies for Sequencing Challenging Samples
                            by seqadmin


                            Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                            03-22-2024, 06:39 AM
                          • seqadmin
                            Techniques and Challenges in Conservation Genomics
                            by seqadmin



                            The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                            Avian Conservation
                            Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                            03-08-2024, 10:41 AM

                          ad_right_rmr

                          Collapse

                          News

                          Collapse

                          Topics Statistics Last Post
                          Started by seqadmin, Yesterday, 06:37 PM
                          0 responses
                          10 views
                          0 likes
                          Last Post seqadmin  
                          Started by seqadmin, Yesterday, 06:07 PM
                          0 responses
                          9 views
                          0 likes
                          Last Post seqadmin  
                          Started by seqadmin, 03-22-2024, 10:03 AM
                          0 responses
                          51 views
                          0 likes
                          Last Post seqadmin  
                          Started by seqadmin, 03-21-2024, 07:32 AM
                          0 responses
                          67 views
                          0 likes
                          Last Post seqadmin  
                          Working...
                          X