Huge NGS data storage and transferring

[himwo]

Hello,

I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

Many Thanks!

Him

[Simon Anders]

If you run your own sequencer, data transfer is a big issue. If you are just a user of sequencing machines and get a dataset every other week or so, I wouldn't worry too much.

First, you wrote "LAN", so I assume your sequencer is in house, and you only need to go through your institute's network. Just do the math: with Gigabit ethernet, a few Gigabyte take less than a minute. If the sequencer is not in house and you send the data through the internet, this may easily become a few hours, which is still fine if you don't need to do this every day.

We work a lot with high-throughput microscopy screens here, and there, the problem gets really severe, as such a screen is easily a terabyte. However, a 1 TB hard disk is less than 100 Euros, and with a bit of bubble wrap, shipping it by mail is no problem. And eSATA docking stations such as this one allow you to plug in an ordinary hard disk without having to open your computer's case.

[cpw]

Originally Posted by himwo

Hello,

I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

Many Thanks!

Him

Can't avoid the truism at all: It depends on the details...
I'd suggest you plan first storage & backup of your (precious) data and than select depending on your budget a way to access this storage. The optimum concerning data throughput is Fibre Channel. In my case, the storage supports FC & my server used for data analysing is within 30m range, therefore I'll buy f-cards (~750€; labeled 8Gb/s but doubt that I will get this rate end-to-end). Gigabit-Ethernet is a cheaper choice which gives a high data rate too. I wouldn't base a workflow on USB, since you'll get a much lower data throughput (in practise I would't expect much more than 200 Mb/s -> 25 MB/s), have to gurantee that the media are at hand and need a backup strategy for the media (means copy your stuff to storage - bad data throughout again)...