SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Long Term Data Storage gendxdoc Bioinformatics 16 01-10-2012 12:45 AM
Data Storage after HiSeq Upgrade sklages Illumina/Solexa 9 06-08-2011 07:48 AM
GALAXY: Huge amount of data? sklages Bioinformatics 1 04-12-2011 05:03 AM
Data Storage Space NGS analyst Bioinformatics 1 01-10-2011 08:22 AM
Data storage rdeborja Bioinformatics 2 11-28-2010 01:46 AM

Reply
 
Thread Tools
Old 03-23-2011, 11:48 PM   #1
himwo
Junior Member
 
Location: Hong Kong

Join Date: Mar 2011
Posts: 1
Default Huge NGS data storage and transferring

Hello,

I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

Many Thanks!

Him
himwo is offline   Reply With Quote
Old 03-24-2011, 12:11 AM   #2
Simon Anders
Senior Member
 
Location: Heidelberg, Germany

Join Date: Feb 2010
Posts: 994
Default

If you run your own sequencer, data transfer is a big issue. If you are just a user of sequencing machines and get a dataset every other week or so, I wouldn't worry too much.

First, you wrote "LAN", so I assume your sequencer is in house, and you only need to go through your institute's network. Just do the math: with Gigabit ethernet, a few Gigabyte take less than a minute. If the sequencer is not in house and you send the data through the internet, this may easily become a few hours, which is still fine if you don't need to do this every day.

We work a lot with high-throughput microscopy screens here, and there, the problem gets really severe, as such a screen is easily a terabyte. However, a 1 TB hard disk is less than 100 Euros, and with a bit of bubble wrap, shipping it by mail is no problem. And eSATA docking stations such as this one allow you to plug in an ordinary hard disk without having to open your computer's case.
Simon Anders is offline   Reply With Quote
Old 03-24-2011, 01:32 AM   #3
cpw
Junior Member
 
Location: Frankfurt/FRG

Join Date: Mar 2011
Posts: 6
Default

Quote:
Originally Posted by himwo View Post
Hello,

I am a new comer for NGS data analysis, and my research team are exploring the hardware configuration for it. As I know, the raw data files are in fastq format which are huge in size (several gigabytes each). I wonder what is the best strategy for storage and transferring the data for analysis. I guess, if I transfer the data through LAN or USB, it still takes a day to transfer 1 sample. It seems not a very effective way, could u please advice?

Besides, I wonder whether the software normally used for NGS data analysis support grid computering?

Many Thanks!

Him
Can't avoid the truism at all: It depends on the details...
I'd suggest you plan first storage & backup of your (precious) data and than select depending on your budget a way to access this storage. The optimum concerning data throughput is Fibre Channel. In my case, the storage supports FC & my server used for data analysing is within 30m range, therefore I'll buy f-cards (~750; labeled 8Gb/s but doubt that I will get this rate end-to-end). Gigabit-Ethernet is a cheaper choice which gives a high data rate too. I wouldn't base a workflow on USB, since you'll get a much lower data throughput (in practise I would't expect much more than 200 Mb/s -> 25 MB/s), have to gurantee that the media are at hand and need a backup strategy for the media (means copy your stuff to storage - bad data throughout again)...
cpw is offline   Reply With Quote
Reply

Tags
next gen sequencing

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 06:30 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO