Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Huge NGS data storage and transferring

    Hello,

    I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

    Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

    Many Thanks!

    Him

  • #2
    If you run your own sequencer, data transfer is a big issue. If you are just a user of sequencing machines and get a dataset every other week or so, I wouldn't worry too much.

    First, you wrote "LAN", so I assume your sequencer is in house, and you only need to go through your institute's network. Just do the math: with Gigabit ethernet, a few Gigabyte take less than a minute. If the sequencer is not in house and you send the data through the internet, this may easily become a few hours, which is still fine if you don't need to do this every day.

    We work a lot with high-throughput microscopy screens here, and there, the problem gets really severe, as such a screen is easily a terabyte. However, a 1 TB hard disk is less than 100 Euros, and with a bit of bubble wrap, shipping it by mail is no problem. And eSATA docking stations such as this one allow you to plug in an ordinary hard disk without having to open your computer's case.

    Comment


    • #3
      Originally posted by himwo View Post
      Hello,

      I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

      Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

      Many Thanks!

      Him
      Can't avoid the truism at all: It depends on the details...
      I'd suggest you plan first storage & backup of your (precious) data and than select depending on your budget a way to access this storage. The optimum concerning data throughput is Fibre Channel. In my case, the storage supports FC & my server used for data analysing is within 30m range, therefore I'll buy f-cards (~750€; labeled 8Gb/s but doubt that I will get this rate end-to-end). Gigabit-Ethernet is a cheaper choice which gives a high data rate too. I wouldn't base a workflow on USB, since you'll get a much lower data throughput (in practise I would't expect much more than 200 Mb/s -> 25 MB/s), have to gurantee that the media are at hand and need a backup strategy for the media (means copy your stuff to storage - bad data throughout again)...

      Comment

      Latest Articles

      Collapse

      • seqadmin
        Strategies for Sequencing Challenging Samples
        by seqadmin


        Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
        03-22-2024, 06:39 AM
      • seqadmin
        Techniques and Challenges in Conservation Genomics
        by seqadmin



        The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

        Avian Conservation
        Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
        03-08-2024, 10:41 AM

      ad_right_rmr

      Collapse

      News

      Collapse

      Topics Statistics Last Post
      Started by seqadmin, Yesterday, 06:37 PM
      0 responses
      8 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, Yesterday, 06:07 PM
      0 responses
      8 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-22-2024, 10:03 AM
      0 responses
      49 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-21-2024, 07:32 AM
      0 responses
      66 views
      0 likes
      Last Post seqadmin  
      Working...
      X