Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • HiSeq 2500 : Rawdata Storage (permanent)

    Hi Folks,

    as an academic sequencing service provider we are obligated to store the sequencing "rawdata".

    We are currently defining "rawdata" as the "standard" rundata plus CIFs (for HiSeq devices).
    Amount of data is roughly 3-4TB per run. 80%-85% being CIFs.
    In the past this enabled us to run OLB on lanes/runs to "optimize" basecalling in some cases.

    OLB development has been dropped some time ago, RTA was developed continously.

    So my questions are:

    * What files/data are you (permanently) storing of HiSeq run? And why?
    * Are you storing the data for (potential) re-analysis (OLB?) or just for documentation purposes?
    * Anyone with experience in running OLB 1.9.x on HiSeq data created with current machine software?

    I am just curious :-)

    best,
    Sven

  • #2
    We store a tar copy of the flowcell folder from the sequencer (No CIF's, with amount of sequencing we do that is impractical). In addition we store fastq files after sample demultiplexing for ready retrieval in case customers ask us to re-release their data (generally because of drive failures, on their end).

    Comment


    • #3
      Originally posted by GenoMax View Post
      We store a tar copy of the flowcell folder from the sequencer (No CIF's, with amount of sequencing we do that is impractical). In addition we store fastq files after sample demultiplexing for ready retrieval in case customers ask us to re-release their data (generally because of drive failures, on their end).
      Well, it becomes even impractical with our 4 machines. It works, but it is still impractical.

      We store the fastq results without any time limit. That's ok ..

      Comment


      • #4
        Are you backing the data up to tape?

        With new nearline storage from google that is becoming a tempting option as a replacement for tape (especially with the announcement that netbackup will support nearline storage in v.7.7) at just 1 cent/GB/month.

        Comment


        • #5
          No, we are backing up to disks, jBods.
          Google is not an option as we require the data to be stored on internal servers.

          Comment


          • #6
            Wow that is a lot of data on plain JBOD. What are you doing for ensuring reliability? Or are you storing data on external drives that are then put away in a closet (not continuously available)?

            Comment


            • #7
              Originally posted by GenoMax View Post
              Wow that is a lot of data on plain JBOD. What are you doing for ensuring reliability? Or are you storing data on external drives that are then put away in a closet (not continuously available)?
              That's how we do it. We keep the results data "online" without any time limit (internal) and put the raw rundata on jbods after a few months to be stored offline in a closet as backup :-)

              Comment


              • #8
                Originally posted by sklages View Post
                That's how we do it. We keep the results data "online" without any time limit (internal) and put the raw rundata on jbods after a few months to be stored offline in a closet as backup :-)
                Ah. This is not standard JBOD (something that always stays online).

                Hopefully you have an assistant who can spin the disks up periodically to make sure they are working

                I have had bad experiences with external drives stored in a closet that have refused to spin up.

                Comment


                • #9
                  Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.

                  Comment


                  • #10
                    Originally posted by Brian Bushnell View Post
                    Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.
                    We stored our sanger sequencing and project on tape; nowadays it's too much data, so it is easier to use harddisks.

                    You do not store any raw run data, e.g. BCLs and/or CIFs etc? What if you/your customer chose a wrong index? No chance of re-processing?

                    We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).

                    Comment


                    • #11
                      We generally don't let customers choose indices. Typically, we prepare the libraries, sequence, quality-control, and assemble or otherwise analyze the data to verify the process was successful before archiving. Most of our customers don't want the raw data anyway.

                      It's not entirely clear to me what additional benefit you get from storing anything prior to the fastqs (demultiplexed or not), as long as all the relevant information (lane, bar codes, etc) is in the read headers.

                      Comment


                      • #12
                        We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

                        Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.

                        Comment


                        • #13
                          Originally posted by sklages View Post
                          We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).
                          That is ~ 6 x 5 TB drives every two weeks?

                          Do you get bare drives or full external enclosures? I hope you have a storage room set aside for those drives.

                          Comment


                          • #14
                            Originally posted by dpryan View Post
                            We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

                            Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.
                            Devon: There is a subtle but significant difference. Google's nearline storage supposedly offers access with just a 3-5 second delay (so you could compute on it via Google compute, Edit: Not 100% certain about this). Glacier is truly meant for long term storage.
                            Last edited by GenoMax; 04-14-2015, 03:00 PM.

                            Comment


                            • #15
                              Originally posted by dpryan View Post
                              We're also storing the raw hiseq output to tape.
                              What is the "raw hiseq output" referring to? Does it include CIF files or is this just the "standard" rundata with BCLs etc?

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Strategies for Sequencing Challenging Samples
                                by seqadmin


                                Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                03-22-2024, 06:39 AM
                              • seqadmin
                                Techniques and Challenges in Conservation Genomics
                                by seqadmin



                                The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                Avian Conservation
                                Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                03-08-2024, 10:41 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 06:37 PM
                              0 responses
                              10 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, Yesterday, 06:07 PM
                              0 responses
                              10 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-22-2024, 10:03 AM
                              0 responses
                              51 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-21-2024, 07:32 AM
                              0 responses
                              67 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X