SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
[Important] You can help SEQanswers to get published marcowanger General 0 08-14-2011 07:13 PM
Cufflinks paper published at Nature Biotechnology Cole Trapnell Bioinformatics 4 04-17-2011 02:27 PM
published ngs data and scripts helprhelp General 2 01-06-2011 06:34 AM
Paper question genehunter2 Illumina/Solexa 1 12-13-2010 12:37 AM
BFAST published nilshomer Bioinformatics 16 11-20-2009 03:23 PM

Reply
 
Thread Tools
Old 08-22-2010, 08:32 PM   #1
xinwu
Member
 
Location: Beijing

Join Date: Jul 2010
Posts: 33
Default One question to the new published paper about Myrna

The latest paper about RNA-seq application is "Cloud-scale RNA-sequencing differential expression analysis with Myrna" written by Ben Langmead, etc. It uses Bowtie, R, Bioconductor. I think they are integrated with Hadoop streaming, since they are all binary executable tools. My question is how to install them on all of hadoop nodes in one step or quickly? Manually installation will be a tedious work and not acceptable when you have a big hadoop cluster, say, over 100+ nodes. I just wonder is there a solution to overcome or fix this issue.
Thanks in advance.
BTW, I don't think binary executeables installation in a shared file system like NFS is a good solution.

Last edited by xinwu; 08-22-2010 at 08:38 PM.
xinwu is offline   Reply With Quote
Old 08-22-2010, 10:16 PM   #2
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by xinwu View Post
BTW, I don't think binary executeables installation in a shared file system like NFS is a good solution.
Please clarify the above.
nilshomer is offline   Reply With Quote
Old 08-23-2010, 12:58 AM   #3
xinwu
Member
 
Location: Beijing

Join Date: Jul 2010
Posts: 33
Default

Quote:
Originally Posted by nilshomer View Post
Please clarify the above.
OK, I should state that "not a good solution for me" rather than "not a good solution" because of some internal limitations.
But NFS itself has limitations if you considerate performance, etc.
xinwu is offline   Reply With Quote
Old 08-23-2010, 06:27 AM   #4
Ben Langmead
Senior Member
 
Location: Baltimore, MD

Join Date: Sep 2008
Posts: 200
Default

Quote:
Originally Posted by xinwu View Post
Manually installation will be a tedious work and not acceptable when you have a big hadoop cluster, say, over 100+ nodes. I just wonder is there a solution to overcome or fix this issue.
Thanks in advance.
BTW, I don't think binary executeables installation in a shared file system like NFS is a good solution.
Hi Xinwu,

I'm not sure why a shared filesystem isn't a good solution -- it works for me, and is way more convenient than installing on each node. As for easy installation: we include a script with Myrna that automatically grabs and builds R and Bioconductor with all of the appropriate packages installed ($MYRNA_HOME/R/build_r.sh). The ideal situation is: you download the Myrna archive to an NFS share, run that script from one of your cluster nodes, and you're done. R is now runnable on all the nodes, and Myrna automatically tries $MYRNA_HOME/R when looking for an appropriate R_HOME.

Also note that if you run in cloud mode (on EMR), R is downloaded to each EC2 node automatically with no user intervention.

Thanks,
Ben
Ben Langmead is offline   Reply With Quote
Old 08-23-2010, 06:33 PM   #5
xinwu
Member
 
Location: Beijing

Join Date: Jul 2010
Posts: 33
Default

Hi Ben,
First, I really appreciate your excellent job on NGS. Actually, I met a potential NFS bug in my computing environment and can not build/install any binary software. I am also not authorized to update Linux/NFS to fix the issue. I apologize if I mislead you by "not a good solution", what I want to say is "is not a choice for me". In my situation, I prefer to install myrna on native system rather than NFS due to the issue I met and some concern of performance, security policy, etc. You said when myrna is run in cloud mode(EMR), R is installed on EACH EC2 node. It indicates cloud mode also does not use shared NFS. The package is actually installed on the native system of every node, right? How do you do that? Or it is done automatically by Amozon Infrastructure? Thanks.
xinwu is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 06:41 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO