Hadoop for human genome data

smylalwys · 01-14-2015, 11:23 PM

Hello Everyone,

How do we store the human genome data using Hadoop (chromosome level) so that we can perform processing (bio-algorithm computing) on the data using Hadoop clusters?

dpryan · 01-15-2015, 12:18 AM

How one best stores the data is entirely dependent on how the actual cluster is constructed and what the nature of the algorithm is. If the cluster is essentially a cloud with slow IO then you'll approach this differently than with a HPC cluster with a faster local storage array. Also, if you just need to load the genome into memory for long computations then it doesn't really matter how you store it, that's not going to be the bottleneck.

smylalwys · 01-15-2015, 12:24 AM

Hi Ryan,
Thanks for your reply. We do have cluster of 30 machines with hadoop. The problem is we are planning to process the human genome project using hadoop. Here the data is in the form of BAM files. I know if I load the data to hdfs, it will automatically split it into chunks and store on the name nodes. Thats is the problem here. I couldn't split the data like that. Need to split the data chromosome wise so that we can perform bio algorithm computing on them.

Can someone please give some insights on this

dpryan · 01-15-2015, 12:26 AM

Without knowing more detail it's impossible to give any guidance. Hadoop is a general tool to facilitate processing. How you should split things depends entirely on what you want to do with the results (and "bio algorithm computing" has absolutely no meaning).

smylalwys · 01-15-2015, 12:32 AM

Bio algorithm computing : for instance bisulfite methylation extraction

dpryan · 01-15-2015, 12:33 AM

Yes, that's one of many possible but completely unrelated tasks. I've already responded to this on one of your biostars threads. Please don't cross post.

smylalwys · 01-15-2015, 12:34 AM

Currently we use bismap ( python tool ). Is there a way to store the data chromosome wise on hadoop.and run the bismap tool command as map reduce jobs

dpryan · 01-15-2015, 12:38 AM

How this would be done would depend entirely on the cluster, but there's generally no single command (or simple series thereof) that would allow that. The traditional way to do this would be to simply tell BSMap's methylation extractor to just process a single chromosome (and then run that simultaneously with different chromosomes on different cores). You could simply do that in a fraction of the time it's take to implement a full hadoop-based solution.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Human Genome Data Variation?	wilson90	Bioinformatics	1	08-31-2012 07:34 AM
Whole-genome sequencing data offer insights into human demography	KerryOdair	Personalized Genomics	0	03-07-2012 11:05 AM
human genome data	gsgs	General	0	06-11-2010 10:07 PM

01-14-2015, 11:23 PM	#1
smylalwys Junior Member Location: indiana Join Date: Jan 2015 Posts: 4	Hadoop for human genome data Hello Everyone, How do we store the human genome data using Hadoop (chromosome level) so that we can perform processing (bio-algorithm computing) on the data using Hadoop clusters?

01-15-2015, 12:18 AM	#2
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	How one best stores the data is entirely dependent on how the actual cluster is constructed and what the nature of the algorithm is. If the cluster is essentially a cloud with slow IO then you'll approach this differently than with a HPC cluster with a faster local storage array. Also, if you just need to load the genome into memory for long computations then it doesn't really matter how you store it, that's not going to be the bottleneck.

01-15-2015, 12:24 AM	#3
smylalwys Junior Member Location: indiana Join Date: Jan 2015 Posts: 4	Hi Ryan, Thanks for your reply. We do have cluster of 30 machines with hadoop. The problem is we are planning to process the human genome project using hadoop. Here the data is in the form of BAM files. I know if I load the data to hdfs, it will automatically split it into chunks and store on the name nodes. Thats is the problem here. I couldn't split the data like that. Need to split the data chromosome wise so that we can perform bio algorithm computing on them. Can someone please give some insights on this

01-15-2015, 12:26 AM	#4
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	Without knowing more detail it's impossible to give any guidance. Hadoop is a general tool to facilitate processing. How you should split things depends entirely on what you want to do with the results (and "bio algorithm computing" has absolutely no meaning).

01-15-2015, 12:32 AM	#5
smylalwys Junior Member Location: indiana Join Date: Jan 2015 Posts: 4	Bio algorithm computing : for instance bisulfite methylation extraction

01-15-2015, 12:33 AM	#6
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	Yes, that's one of many possible but completely unrelated tasks. I've already responded to this on one of your biostars threads. Please don't cross post.

01-15-2015, 12:34 AM	#7
smylalwys Junior Member Location: indiana Join Date: Jan 2015 Posts: 4	Currently we use bismap ( python tool ). Is there a way to store the data chromosome wise on hadoop.and run the bismap tool command as map reduce jobs

01-15-2015, 12:38 AM	#8
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	How this would be done would depend entirely on the cluster, but there's generally no single command (or simple series thereof) that would allow that. The traditional way to do this would be to simply tell BSMap's methylation extractor to just process a single chromosome (and then run that simultaneously with different chromosomes on different cores). You could simply do that in a fraction of the time it's take to implement a full hadoop-based solution.