SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
multithreaded GATK local realignment adaptivegenome Open Genomics Engine Project 1 08-30-2012 08:46 AM
Where is Casava's Demux Multithreaded? lednakashim Bioinformatics 2 06-25-2012 04:20 PM
Help! submit jobs about Corona Lite skblazer SOLiD 16 05-12-2010 07:43 AM
Jobs forum structure changed. ECO Site Announcements 0 11-12-2009 10:55 PM
jobs forum pmcget Site Feedback/Suggestions 1 01-22-2008 09:56 AM

Reply
 
Thread Tools
Old 04-29-2013, 12:27 PM   #1
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default multithreaded jobs

I got a question from our cluster IT that I could not answer. I am having problems runing BWA, samtools or blastx in the multithreaded mode on the cluster. Cluster resorces are normally reserved by_core, that is multithreaded jobs are spread across multiple nodes. BWA, samtools and blast+ run fine on my 2 Quad Core CPU with hyperthreading, that is with 16 threads. The question was - can actually BWA, samtools and blastx+ run with multiple threads when spread across several nodes? If not, this answers the question. If yes, are there any specifics/pecularities in scheduling resources?
yaximik is offline   Reply With Quote
Old 04-29-2013, 12:41 PM   #2
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,087
Default

Depending on what scheduling software your cluster uses you can specify multiple threads along with an equivalent number of cores (-n in LSF) to go with them.

That said you would want to be judicious when spreading jobs across cores/nodes. Depending on what else may be running on a particular server/node you may overload that server/node (which your admins may not like). There may also be some I/O problems if you do not have a high performance storage system that is available on the cluster and your jobs get spread across several nodes.

In case of LSF you can use flags (-x) that can exclusively reserve a node for your use with multiple cores which would ensure you do not step on someone else' jobs. There are probably equivalent examples for SGE.
GenoMax is offline   Reply With Quote
Old 04-29-2013, 01:11 PM   #3
Khen
Member
 
Location: Las Vegas

Join Date: Mar 2012
Posts: 11
Default

Definitely need to find out what your scheduling software is. But in general, I prefer parallelization to multithreading. It is much easier and I find that I run into fewer problems.
Khen is offline   Reply With Quote
Old 04-29-2013, 01:21 PM   #4
rhinoceros
Senior Member
 
Location: sub-surface moon base

Join Date: Apr 2013
Posts: 372
Default

Much better than running a single multithreaded instance of blast over many nodes is to split your input (e.g. with fastasplitn) and then run multiple instances of blast so that the maximum number of threads of each instance equals the number of cores of one CPU. With SGE this is achieved by using the smp parallel environment. You can make a simple bash script, e.g.:

Code:
cat blastp.sh

#!/bin/bash
#$ -N blastp
#$ -j y
#$ -cwd
#$ -pe smp 8
#$ -R y
blastp -query input.${SGE_TASK_ID} -db nr -lotsOfFlags -outfmt 6 -num_threads 8 -out ${SGE_TASK_ID}.tsv
And call it:

qsub -t 1-10:1 blastp.sh

To start/queue 10 instances of blastp (each with 8 threads). Input for this would be input.1, input.2, .., input.10 and you could just concatenate the results with cat..

Last edited by rhinoceros; 04-29-2013 at 01:36 PM.
rhinoceros is offline   Reply With Quote
Old 04-29-2013, 04:42 PM   #5
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by rhinoceros View Post
Much better than running a single multithreaded instance of blast over many nodes is to split your input (e.g. with fastasplitn) and then run multiple instances of blast so that the maximum number of threads of each instance equals the number of cores of one CPU. With SGE this is achieved by using the smp parallel environment. You can make a simple bash script, e.g.:

Code:
cat blastp.sh

#!/bin/bash
#$ -N blastp
#$ -j y
#$ -cwd
#$ -pe smp 8
#$ -R y
blastp -query input.${SGE_TASK_ID} -db nr -lotsOfFlags -outfmt 6 -num_threads 8 -out ${SGE_TASK_ID}.tsv
And call it:

qsub -t 1-10:1 blastp.sh

To start/queue 10 instances of blastp (each with 8 threads). Input for this would be input.1, input.2, .., input.10 and you could just concatenate the results with cat..
Yep I thought about that, but I was not sure that it is more or as efficient as multithreading. blasp is legacy blast, is not it? If so, one has no choice, as multithreading was intriduced with blast+, was not it? Anyway, good to know it works well, thanks!
yaximik is offline   Reply With Quote
Old 04-29-2013, 04:48 PM   #6
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by Khen View Post
Definitely need to find out what your scheduling software is. But in general, I prefer parallelization to multithreading. It is much easier and I find that I run into fewer problems.
Well, that is a separate interesting question. GNU parallel is one way to go, but I was not yet quite able figuring out how to run it on a cluster. I am not a programmer, I mean I do not have sufficent exerience yet, so how would you request resources for GNU parallel? Unless you parallelize in some other way...
yaximik is offline   Reply With Quote
Old 04-30-2013, 04:15 AM   #7
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,087
Default

Quote:
Originally Posted by yaximik View Post
Well, that is a separate interesting question. GNU parallel is one way to go, but I was not yet quite able figuring out how to run it on a cluster. I am not a programmer, I mean I do not have sufficent exerience yet, so how would you request resources for GNU parallel? Unless you parallelize in some other way...
Can we stick to the original question you had posted? What scheduling software is the cluster you have access to running? On a cluster you are going to get the most benefit by using the job scheduling system.

Do I sense some resistance on your side to give up on your dedicated workstation (where you are in control)? In the long run using a dedicated cluster would allow you to get much more done.

Last edited by GenoMax; 04-30-2013 at 04:35 AM.
GenoMax is offline   Reply With Quote
Old 04-30-2013, 04:54 AM   #8
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

GNU parallel is not designed for clusters. LSF and SGE/UGE are.
lh3 is offline   Reply With Quote
Old 04-30-2013, 06:43 AM   #9
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by GenoMax View Post
Can we stick to the original question you had posted? What scheduling software is the cluster you have access to running? On a cluster you are going to get the most benefit by using the job scheduling system.
Oh, that was not intention to swing away, I was just not sure how parallelization was achived. Our grid (I call it cluster incorrectly) uses SGE. Our documentation is very sketchy, so I have to browse a lot of docs posted for other grids or ask silly questions if I cannot find answer. Also, local implementation differ, so if I happen to find a seemengly usable script it often misunderstood by our scheduler.

Quote:
Do I sense some resistance on your side to give up on your dedicated workstation (where you are in control)? In the long run using a dedicated cluster would allow you to get much more done.
I use my server to test multithreaded jobs, so I won't alienate IT by crashing nodes on the grid. I certainly would prefer to get a dedicated cluster equipped with a bunch of GPUs, but it just takes time to get funds for that. At current 8% payline at NIH it is quite challenging. So I have now only my server and the local SGE grid at my disposal.
yaximik is offline   Reply With Quote
Old 04-30-2013, 06:47 AM   #10
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by lh3 View Post
GNU parallel is not designed for clusters. LSF and SGE/UGE are.
Looks like I may have hit a few wrong buttons in message window. if so I apologize. So GNU parallel can be used with SGE? I asked this question in separate thread earlier but got no answers.
yaximik is offline   Reply With Quote
Old 04-30-2013, 07:53 AM   #11
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

Parallel simply launches multiple jobs on one machine. In principle, you can launch parallel inside qsub/bsub. On LSF, it should be something like: bsub -n 8 'parallel -j 8 ...'. However, with LSF/SGE, this is a bad idea. You should just let LSF/SGE manage all your jobs. Parallel has little use except for constructing command lines. As GenoMax has suggested, just use SGE. You won't crash nodes by submitting multithreaded jobs in a small batch.

Common use of SGE (such as specifying #cpus) works everywhere. If your system admins set resource limits on queues, ask them. We can offer little help.

For most NGS analyses, investing on GPU is a waste of money before you really understand how that works and/or what programs are available.
lh3 is offline   Reply With Quote
Old 04-30-2013, 07:58 AM   #12
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,087
Default

So we now know that you are going to use SGE.

Can you take a moment and clarify if you are using a true "grid" or a compute "cluster". Here is a posting that would help with the definitions.

If you are truly using a "grid", meaning compute resources that are connected over a non-local network, then trying to split/multi-thread jobs would be a bad idea.
GenoMax is offline   Reply With Quote
Old 04-30-2013, 02:04 PM   #13
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by GenoMax View Post
So we now know that you are going to use SGE.

Can you take a moment and clarify if you are using a true "grid" or a compute "cluster". Here is a posting that would help with the definitions.

If you are truly using a "grid", meaning compute resources that are connected over a non-local network, then trying to split/multi-thread jobs would be a bad idea.
From that definitions it is a grid. It is likely located somewhere on the campus, composed of nodes with at least three different architectures and connceted by InfiniBand network. AT least they officially call it a grid.

It looks like I already have learned the answer in a hard way. My last 64 and 96 thread blastx jobs died with the following error:

Code:
A daemon (pid 19390) died unexpectedly with status 137 while attempting
to launch so we are aborting.
There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.

mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
The scheduling script is below:
Code:
#$ -S /bin/bash
#$ -cwd
#$ -N SC3blastx_64-96thr
#$ -pe openmpi* 64-96
#$ -l h_rt=24:00:00,vf=3G
#$ -j y
#$ -M yaximik@gmail.com
#$ -m eas
#
# Load the appropriate module files
# Should be loaded already
#$ -V

mpirun -np $NSLOTS blastx -query myquery.fasta -db nr -out query.out -evalue 0.001 -max_intron_length 100000 -outfmt 5 -num_alignments 20 -lcase_masking -num_threads $NSLOTS
I asked grid IT and they said they had to kill it as the job was overloading nodes. They saw loads up to 180 instead of close to 12 on 12-core nodes. They think that blastx is not an openmpi application, so openMPI is spawning between 64-96 blastx processes, each of which is then starting up 96 worker threads. Or if blastx can work with openmpi, my blastx synthax mpirun syntax is wrong. Is this correct?

I was advised earlier by someone from openmpi user group to use –pe openmpi [ARG} , where ARG = number_of_processes x number_of_threads , and then pass desired number of threads as ‘ mpirun –np $NSLOTS cpus-per-proc [ number_of_threads]’. When I did that, I got an error that more threads were requested than number of physical cores.

Oh,well... Looks like splitting large datafiles to thousands of smaller pieces is the only option to use blastx. Any other suggestions?
yaximik is offline   Reply With Quote
Old 04-30-2013, 06:15 PM   #14
lh3
Senior Member
 
Location: Boston

Join Date: Feb 2008
Posts: 693
Default

As rhinoceros has suggested, you should split your input files into small batches. Blast+ is not aware of MPI. There is a mpiblast, but as I remember, it is not blast+.
lh3 is offline   Reply With Quote
Old 04-30-2013, 06:34 PM   #15
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 7,087
Default

Here are two useful pages:

http://www3.imperial.ac.uk/bioinfsup...ing_array_jobs
http://www3.imperial.ac.uk/bioinfsup..._parallel_jobs
GenoMax is offline   Reply With Quote
Old 05-01-2013, 06:29 AM   #16
krobison
Senior Member
 
Location: Boston area

Join Date: Nov 2007
Posts: 747
Default

Quote:
Originally Posted by yaximik View Post
:The question was - can actually BWA, samtools and blastx+ run with multiple threads when spread across several nodes? If not, this answers the question. If yes, are there any specifics/pecularities in scheduling resources?
I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.

As other posters have noted, what you would need to do for BLAST/BWA/bowtie/samtools etc is split your job into sub-jobs, run those on the cluster using the cluster scheduling software, and then merge the results at the end.
krobison is offline   Reply With Quote
Old 05-01-2013, 07:23 AM   #17
rhinoceros
Senior Member
 
Location: sub-surface moon base

Join Date: Apr 2013
Posts: 372
Default

Quote:
Originally Posted by krobison View Post
I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.

As other posters have noted, what you would need to do for BLAST/BWA/bowtie/samtools etc is split your job into sub-jobs, run those on the cluster using the cluster scheduling software, and then merge the results at the end.
SGE's orte parallel environment can split blast threads over multiple nodes. However, using blast like this is a waste of resources as most of the time blast isn't heavy on the CPU(s) anyway and infiniband is a dog in comparison to smp. Thus, far faster is to run multiple multithreaded instances of blast with split input (not thousands of files, but some 10-50). Our newest cluster has 19 nodes, 2 x 8 cores in 17 nodes with 256 GB ram each and 2 nodes with 2 x 16 cores and 756 GB ram each. In this setup, I've been blasting predicted proteins from metagenomes (100s of thousands to more than a million peptides) against nr in very reasonable times with a script similar to what I mentioned earlier. I don't know it it's 'the best' way to do it, but I know it works..

p.s. I'm not sure blast xml output(s) can be concatenated like the tabular output. Additionally, the xml output files grow ridiculously large very fast. Additional thing to consider: run your blasts from a directory that doesn't get backed up on a regular basis (usually something like /jobs/youruid).

Last edited by rhinoceros; 05-01-2013 at 07:59 AM.
rhinoceros is offline   Reply With Quote
Old 05-01-2013, 09:29 AM   #18
yaximik
Senior Member
 
Location: Oregon

Join Date: Apr 2011
Posts: 205
Default

Quote:
Originally Posted by krobison View Post
I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.
I guess I was confused by FAQ entry from open-mpi.org
http://www.open-mpi.org/faq/?categor...n-mpi-programs
While uptime in the example is certainly not specifically openmpi-enabled, it is not multithreaded per se as, say blastx, so event if multiple instances of it can be launched with mpirun, attempts to use multithreading creates a lot of mess.
yaximik is offline   Reply With Quote
Old 05-01-2013, 10:10 AM   #19
rhinoceros
Senior Member
 
Location: sub-surface moon base

Join Date: Apr 2013
Posts: 372
Default

Quote:
Originally Posted by yaximik View Post
I guess I was confused by FAQ entry from open-mpi.org
http://www.open-mpi.org/faq/?categor...n-mpi-programs
While uptime in the example is certainly not specifically openmpi-enabled, it is not multithreaded per se as, say blastx, so event if multiple instances of it can be launched with mpirun, attempts to use multithreading creates a lot of mess.
I'm not 100% sure, but I think your script simply starts the same multithreaded job multiple times. Blast is not mpi compatible.
rhinoceros is offline   Reply With Quote
Old 05-02-2013, 01:48 AM   #20
dpryan
Devon Ryan
 
Location: Freiburg, Germany

Join Date: Jul 2011
Posts: 3,480
Default

Quote:
Originally Posted by rhinoceros View Post
I'm not 100% sure, but I think your script simply starts the same multithreaded job multiple times. Blast is not mpi compatible.
I'm ~99% sure you're correct. Starting the same non-mpi-aware process on multiple nodes of a grid or cluster will just cause problems. The various instances won't talk to each other, which is what yaximik is hoping will happen.
dpryan is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 03:09 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2021, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO