SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Encore NGS Library Systems for Ion Torrent from NuGEN Technologies NuGEN Vendor Forum 1 01-13-2015 09:02 AM
Best deployment method of NGS tools on a cluster? greener Bioinformatics 4 02-16-2012 05:15 AM
Texas AgriLife: NGS Systems Analyst Position CharlieJohnson Academic/Non-Profit Jobs 0 10-12-2011 09:12 AM
454 and cluster operating systems - patching Fedora rglover 454 Pyrosequencing 3 05-20-2010 07:23 AM

Reply
 
Thread Tools
Old 09-10-2011, 01:57 AM   #1
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Question GPU/Cluster/Server based systems for NGS

Hi all,
I am new to NGS field but I do understand some of the things and now want to know about better hardware required for NGS data analysis.
Please explain with respect to categories
1) Whole genome sequencing
2) Exome sequencing
3) Targeted sequencing

My question particularly is which hardware is good for NGS data analysis?
Tesla GPU OR Servers OR Compute cluster?

Now a days GPU are becoming of much interest but do anyone have experience of GPU in NGS analysis and if yes please share it.
Servers with multiple cores are sufficient or a cluster of 10-20 nodes?
Also explain about cost/performance in each case.
PratikC is offline   Reply With Quote
Old 09-10-2011, 07:48 AM   #2
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

As to GPU computing, you would like to check out this post first:

http://seqanswers.com/forums/showthread.php?t=13497
lh3 is offline   Reply With Quote
Old 09-11-2011, 10:29 AM   #3
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Default

I already checked that. But mummer and ugene are ported to GPGPU and my question is not limited to GPU but a comparison like workstation/server having multi core processor and single/dual HDD will be having problem because of extensive I/O in single HDD while clusters will have HDD in each node so will not face I/O problem but then we need software to be compatible with MPI. While GPU is very much faster but it will also face problem of I/O like server/workstation. So I just wanted some people's opinion and wanted to know about their experience in different platforms.
PratikC is offline   Reply With Quote
Old 09-12-2011, 05:07 AM   #4
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,079
Default

There is no single answer for a question such as this except saying "it would depend on what exactly you want to do".

From my experience, you can't go wrong with a small cluster (< 10 nodes) for doing all of the analyses that you have mentioned (if you need to do "de novo" assemblies of large genomes then that may require special consideration).

You would need to develop some patience depending on what hardware you choose since something may take an hour or two more to finish if you went with slightly slower CPU's/less RAM.

You have mentioned having a disk or two in each node but you would not want to move files in and out to these since that will cause a major overhead specially if you intend to analyze hundreds of samples. Your storage subsystem will be become important if you are going to go with a cluster since your processing will likely become I/O bound beyond a certain number of simultaneous processes. You would also want to invest in a better switch that would be tying all this together.
GenoMax is offline   Reply With Quote
Old 09-12-2011, 05:41 AM   #5
gprakhar
Member
 
Location: India

Join Date: Aug 2010
Posts: 78
Default

1) For GPU usage you obviously need algorithms that have been ported for GPU. And the list as you know is very small. We have a small test set of Tesla GPUs, but nothing to rum on that.
2) If you are looking at human Whole Genome alignment, then at least 10 Nodes are required to do that, that too if the Aligner is MPI enabled.
3) We started of 9 months ago with 20 Nodes with 48Gb RAM and Intel Xeon 12core CPU. Had to upgrade to 30 Nodes in less than six months, Same for storage, started with 25Tb and now upgraded it 50Tb.
4) Still 90% of the time the cluster is working on full capacity.
5) The storage has to be on the Network, a distributed Storage system will be a good idea. Panasas is a good choice. Its MPI enabled.
6) Also you would need Tape storage to archive the raw and analyzed data, from time to time.
gprakhar is offline   Reply With Quote
Old 09-12-2011, 07:57 AM   #6
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Default

Thanx GenoMax and gprakhar.
I understand that GPU is not suitable for NGS data analysis at this point because no good software ported to GPU.
About cluster, what you people suggest is for network drive. I was thinking dedicated drive in each node will be better but as per your suggestion, I checked Panasas and seems to be good.
We are going to analyze 100s of samples of human whole exome. So if you have any idea about the computational requirement, that will be very useful.

For cluster, I have one more question, has anyone compiled NGS analysis software with MPI? I googled for bowtie and other software compiled with MPI but didnt found anything good. So if you have any suggestions for MPI enabled software, that will be much appreciated.
PratikC is offline   Reply With Quote
Old 09-12-2011, 08:37 AM   #7
rskr
Senior Member
 
Location: Santa Fe, NM

Join Date: Oct 2010
Posts: 250
Default

Its not worth writing GPU/FPGA oriented code for general bioinformatcs, what we really want are mpi code for clusters. The problem with single chip code is that generally the code doesn't scale, so while sometimes it can get great speedups, for a gigabase of data, when you have an exabase you still have to use a cluster, meanwhile there are many embarrassing parallel applications that run perfectly well on clusters, that don't take advantage of single chip speedups.

There might be applications for FPGA/GPU type code in the sequencing machines themselves, since they are pushing the limits of what an on board signal analysis pipeline can do, and need to be tightly coupled to the sequencer itself, and can't be disrupted by the eccentricities of a cluster, and aren't expected to do a wide range of computation.
rskr is offline   Reply With Quote
Old 09-12-2011, 08:59 AM   #8
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

For many NGS applications, I do not see much use of MPI. Many NGS tasks can be naturally parallelized by simply splitting reads into several chunks and then launching separate jobs on the computing nodes. I agree supporting MPI would be more convenient, but I do not think that deserves the extra development time. Also, MPI is not available everywhere. Configurations alone will push some users away. In the few genome centers I am/was working at, MPI supports are also very limited. These centers happily process terabytes of data every week across hundreds nodes without MPI at all.

MPI is only necessary for a few applications (such as some de novo assembly algorithms), but other than those it increases development time for little reward. The lack of MPI-supported tools itself implies my speculation is likely to be true.

Last edited by lh3; 09-12-2011 at 09:07 AM.
lh3 is offline   Reply With Quote
Old 09-12-2011, 09:07 AM   #9
rskr
Senior Member
 
Location: Santa Fe, NM

Join Date: Oct 2010
Posts: 250
Default

Quote:
Originally Posted by lh3 View Post
For many NGS applications, I do not see much use of MPI. Many NGS tasks can be naturally parallelized by simply splitting reads into several chunks and then launching separate jobs on the computing nodes. I agree supporting MPI would be more convenient, but I do not think that deserves the extra development time. Also, MPIs are not available everywhere. Configurations alone will push some users away. In the few genome centers I am/was working at, MPI supports are also very limited. These centers happily process terabytes of data every week without MPI at all.

MPI is only necessary for a few applications (such as some de novo assembly algorithms), but other than those it increases development time for little reward. The lack of MPI-supported tools itself implies my speculation is likely to be true.
I said between MPI and embarrassing parallel jobs, there is little that GPU's can bring to the party. If you aren't aware, embarrassing parallel refer to jobs that can be "chunked" and run on separate processors with no interaction.
rskr is offline   Reply With Quote
Old 09-12-2011, 09:11 AM   #10
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

@rskr: I was not replying to you. I was replying to PratikC about why he sees so few MPI programs and why he should not rely on MPI programs. The thread I pointed out above has already concluded the discussion related to GPU computing.

Last edited by lh3; 09-12-2011 at 09:13 AM.
lh3 is offline   Reply With Quote
Old 09-12-2011, 11:09 AM   #11
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Default

Thanks rskr and lh3. I understood the limitation of GPU at this time, may be in near future there will be many other software ported to GPU but right now cluster is better.
I still dont understand difference between using MPI and running job by spliting. What do I understand is if we have source code, we can compile executable using MPI library and we do have bowtie source code. And if we can not compile bowtie using MPI than how to split job and run them on cluster, I mean how do we distribute job in each node from a single terminal?
If that task is tedious than should we go for multi processor workstation? And what if we have a workstation with 2 or more Intel Xeon processors(quad core), do we need to do anything to utilize all cores for bowtie?
PratikC is offline   Reply With Quote
Old 09-12-2011, 01:07 PM   #12
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,079
Default

Have a look at this site: http://www.rocksclusters.org/wordpress/ This is a popular cluster OS distribuition.

Jobs are generally handled by a "job queuing system" (SGE (sun grid engine) or PBS are available for "rocks") in a cluster. There are commercial job schedulers (e.g. LSF from platform computing) that are generally pretty expensive.

Some of the aligners will allow you to use multiple threads for single jobs but it may be easiest to go the route of starting multiple alignment jobs in parallel for multiple files (or for a single sequence file split into pieces). You will need to combine and reconcile the multiple BAM files if you choose to split the original file.

Quote:
Originally Posted by PratikC View Post
I mean how do we distribute job in each node from a single terminal?
If that task is tedious than should we go for multi processor workstation? And what if we have a workstation with 2 or more Intel Xeon processors(quad core), do we need to do anything to utilize all cores for bowtie?
GenoMax is offline   Reply With Quote
Old 09-12-2011, 08:29 PM   #13
zee
NGS specialist
 
Location: Malaysia

Join Date: Apr 2008
Posts: 249
Default

We happily process NGS data using MPI and/or a scheduling system. The two work quite well if the configuration is done correctly.
Go for simple commodity hardware in the form of boxes or blade servers, depending on how many lanes/plates you need to process per week. A 96-200 core system will be sufficient to support most non-denovo assembly work for large eukaryote genomes. But with the cost of memory diminishing 96Gb per node is nice to have.
Be sure to have enough storage available for your analysis work. We usually recommend a minimum of 12Tb for small labs supporting 1 Illumina sequencer.
zee is offline   Reply With Quote
Old 09-13-2011, 02:20 AM   #14
gprakhar
Member
 
Location: India

Join Date: Aug 2010
Posts: 78
Default

Quote:
Originally Posted by PratikC View Post
I still dont understand difference between using MPI and running job by spliting. What do I understand is if we have source code, we can compile executable using MPI library and we do have bowtie source code. And if we can not compile bowtie using MPI than how to split job and run them on cluster, I mean how do we distribute job in each node from a single terminal?
Just having a MPI library and the source code of the program is not enough. The program has to written using MPI functions. So eg, Bowtie which is not MPI enabled can not be run on multiple nodes. Hence for a big input, divide the total number of reads into 'n' parts then run each part as a separate input to Bowtie, on different Nodes.
But If you have something like Novoalign MPI, which is MPI enabled, this aligner can be run on multiple Nodes. We use it for most of our Alignment related work, it accurate and very fast. But its a commercial software.
Hence to use MPI, the dependency is the code and not the computing environment. As for using Bowtie with MPI, someone will have to re-write the code.

Quote:
If that task is tedious than should we go for multi processor workstation? And what if we have a workstation with 2 or more Intel Xeon processors(quad core), do we need to do anything to utilize all cores for bowtie?
Yes that will give you a speed up. On a single node with multiple cores, run Bowtie with the one part of the split input. Hence for 'n' inputs run on 'n' nodes. On each Node use all the cores available. At the end merge all the files using eg. samtools

Regards
--
pg
gprakhar is offline   Reply With Quote
Old 09-13-2011, 07:28 AM   #15
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Smile

Quote:
Originally Posted by GenoMax View Post
Have a look at this site: http://www.rocksclusters.org/wordpress/ This is a popular cluster OS distribuition.

Jobs are generally handled by a "job queuing system" (SGE (sun grid engine) or PBS are available for "rocks") in a cluster. There are commercial job schedulers (e.g. LSF from platform computing) that are generally pretty expensive.

Some of the aligners will allow you to use multiple threads for single jobs but it may be easiest to go the route of starting multiple alignment jobs in parallel for multiple files (or for a single sequence file split into pieces). You will need to combine and reconcile the multiple BAM files if you choose to split the original file.
Thanks GenoMax.
I checked rocks, its really very good. I will try it out ASAP!

Quote:
Originally Posted by zee View Post
We happily process NGS data using MPI and/or a scheduling system. The two work quite well if the configuration is done correctly.
Go for simple commodity hardware in the form of boxes or blade servers, depending on how many lanes/plates you need to process per week. A 96-200 core system will be sufficient to support most non-denovo assembly work for large eukaryote genomes. But with the cost of memory diminishing 96Gb per node is nice to have.
Be sure to have enough storage available for your analysis work. We usually recommend a minimum of 12Tb for small labs supporting 1 Illumina sequencer.
Thanks zee,
as I said we are not doing whole genome analysis but exome and targeted sequencing and so I am thinking to start with 48 GB/node and 8 TB HDD. Please correct me if I am underestimating the work load of exome.

Quote:
Originally Posted by gprakhar View Post
Just having a MPI library and the source code of the program is not enough. The program has to written using MPI functions. So eg, Bowtie which is not MPI enabled can not be run on multiple nodes. Hence for a big input, divide the total number of reads into 'n' parts then run each part as a separate input to Bowtie, on different Nodes.
But If you have something like Novoalign MPI, which is MPI enabled, this aligner can be run on multiple Nodes. We use it for most of our Alignment related work, it accurate and very fast. But its a commercial software.
Hence to use MPI, the dependency is the code and not the computing environment. As for using Bowtie with MPI, someone will have to re-write the code.


Yes that will give you a speed up. On a single node with multiple cores, run Bowtie with the one part of the split input. Hence for 'n' inputs run on 'n' nodes. On each Node use all the cores available. At the end merge all the files using eg. samtools

Regards
--
pg
Thanks gprakhar,
I got your point. So for bowtie, spliting job and using scheduler will be suitable.

PratikC is offline   Reply With Quote
Old 09-14-2011, 08:09 AM   #16
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Default

Guys, what do you say about cloud computing? If anyone has experience of clouds, please suggest which one is better in terms of cost and processing? We have got 10 Mbps dedicated line and we are going to do human exome sequencing not whole genome.

If anyone knows good resources for tutorial/guide line to run NGS analysis on cluster and or cloud, please tell me because doing this IT related stuff only on my own is little bit difficult for me.
Thanks in advance.
PratikC is offline   Reply With Quote
Old 09-14-2011, 11:00 AM   #17
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,079
Default

If you are going to use a cloud provider like amazon then remember that you are going to pay for getting data in to the cloud (and then getting the results out). If you are planning to store that data then figure that cost in as well. If your ISP charges you for the data based on traffic volume going over that dedicated line then the costs will add up rapidly.

A 50 bp single end HiSeq Illumina sequence data file can be 20-30 GB (even if you multiplex, the file size per lane will add up to this). Depending on how many samples you are planning to run do the calculation of what you will pay to move (along with the time it till take to move the data) and store that data before considering cloud computing.

There is a nice write-up about running a tool called "Galaxy" in the cloud here: http://wiki.g2.bx.psu.edu/Admin/Cloud. Galaxy has a nice suite of NGS tools built in.

Quote:
Originally Posted by PratikC View Post
Guys, what do you say about cloud computing? If anyone has experience of clouds, please suggest which one is better in terms of cost and processing? We have got 10 Mbps dedicated line and we are going to do human exome sequencing not whole genome.

If anyone knows good resources for tutorial/guide line to run NGS analysis on cluster and or cloud, please tell me because doing this IT related stuff only on my own is little bit difficult for me.
Thanks in advance.
GenoMax is offline   Reply With Quote
Old 09-15-2011, 03:40 AM   #18
PratikC
Member
 
Location: India

Join Date: Sep 2011
Posts: 16
Default

Thank you very much GenoMax, the link you gave is very useful.
PratikC is offline   Reply With Quote
Reply

Tags
cluster computing, gpu, server

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 05:06 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO