Can I use multicore processor for Java applications Picard and GATK? (multi-core)

JoshT · 10-16-2014, 06:59 AM

I am developing a pipeline for calling SNPs in RNAseq data (based on Piskol R, Ramaswami G, Li JB. Reliable identification of genomic variants from RNA-seq data. Am J Hum Genet. 2013 Oct 3;93(4):641-51.).

Now that I am scaling up to my real dataset, I would like my pipeline to run faster. The pipeline involves uses Picard Tools v1.84 (ReorderSam, MarkDuplicates, and BuildBamIndex) and GenomeAnalysisTK v2.3-9-ge5ebf34 (RealignerTargetCreator, IndelRealigner, TableRecalibration, and UnifiedGenotyper). I am working on a computer with 8 cores and 64Gb of memory.

Is there a way to run Picard Tools and GATK on all 8 cores? I've searched for general Java command line options and specific Picard Tools and GATK options, to no avail.

Thanks
Josh

dpryan · 10-16-2014, 09:25 AM

Many of the GATK tools can use multiple cores (see the -nt option). For picard, I think it's mostly single-threaded. Of course, you can also just run picard on individual samples in parallel (likewise with GATK).

adamyao · 10-16-2014, 05:41 PM

Sambamba (http://lomereiter.github.io/sambamba/) provides some useful multi-threaded utilities like view, sort, mark duplicate, etc. We have taken advantage of this to speed up our process.

JoshT · 10-16-2014, 07:09 PM

dpryan, thanks for the link. Picard Tools doesn't support multi-threading for my purposes. But GATK does for some.

JoshT · 10-16-2014, 07:10 PM

adamyao, thanks for the link. I'll check out that program.

Nilaksha · 10-16-2014, 08:46 PM

Just a reminder : you can use java -Xmx[your memory]G to get the maximum use of it. In your case you can use
java -Xmx60G -jar /picard-tools/ ... (Leave 4GB for background processes)

GenoMax · 10-17-2014, 03:06 AM

Intel had presented some work they have done with NGS/GATK but I am not sure if the optimized code is available: http://bioinformatics.gatech.edu/sit...Processing.pdf

Baseless · 10-17-2014, 05:36 AM

On GATK it is possible on many tools to use the -nt or -nct options für using multiple threads. However, my advice is to go through every step with a benchmarking dataset to find where you get the best data per time ratio and maybe rather analyze multiple datasets at once than wase time by using more cores than it is beneficial for each step.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Picard: problem with Java?	aleidenroth	Bioinformatics	9	06-29-2013 07:37 PM
Picard Collect Insert Size Java Problem	chongm	Bioinformatics	0	02-11-2013 11:01 AM
using snappy java in picard tools	doc.ramses	Bioinformatics	3	10-04-2011 07:17 AM
how to check whether a bam fille is sorted using picard in java	jay2008	Bioinformatics	0	05-23-2011 03:14 PM
Picard MarkDuplicates throws a java.lang.NegativeArraySizeException	rdeborja	Bioinformatics	0	01-21-2011 08:28 AM

10-16-2014, 06:59 AM	#1
JoshT Junior Member Location: Houston Join Date: Oct 2014 Posts: 3	Can I use multicore processor for Java applications Picard and GATK? (multi-core) I am developing a pipeline for calling SNPs in RNAseq data (based on Piskol R, Ramaswami G, Li JB. Reliable identification of genomic variants from RNA-seq data. Am J Hum Genet. 2013 Oct 3;93(4):641-51.). Now that I am scaling up to my real dataset, I would like my pipeline to run faster. The pipeline involves uses Picard Tools v1.84 (ReorderSam, MarkDuplicates, and BuildBamIndex) and GenomeAnalysisTK v2.3-9-ge5ebf34 (RealignerTargetCreator, IndelRealigner, TableRecalibration, and UnifiedGenotyper). I am working on a computer with 8 cores and 64Gb of memory. Is there a way to run Picard Tools and GATK on all 8 cores? I've searched for general Java command line options and specific Picard Tools and GATK options, to no avail. Thanks Josh Last edited by JoshT; 10-16-2014 at 10:15 AM. Reason: formatting

10-16-2014, 09:25 AM	#2
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	Many of the GATK tools can use multiple cores (see the -nt option). For picard, I think it's mostly single-threaded. Of course, you can also just run picard on individual samples in parallel (likewise with GATK).

10-16-2014, 05:41 PM	#3
adamyao Member Location: Taiwan Join Date: Feb 2011 Posts: 19	Sambamba (http://lomereiter.github.io/sambamba/) provides some useful multi-threaded utilities like view, sort, mark duplicate, etc. We have taken advantage of this to speed up our process.

10-16-2014, 07:09 PM	#4
JoshT Junior Member Location: Houston Join Date: Oct 2014 Posts: 3	dpryan, thanks for the link. Picard Tools doesn't support multi-threading for my purposes. But GATK does for some.

10-16-2014, 07:10 PM	#5
JoshT Junior Member Location: Houston Join Date: Oct 2014 Posts: 3	adamyao, thanks for the link. I'll check out that program.

10-16-2014, 08:46 PM	#6
Nilaksha Member Location: Sri Lanka Join Date: Mar 2014 Posts: 19	Just a reminder : you can use java -Xmx[your memory]G to get the maximum use of it. In your case you can use java -Xmx60G -jar /picard-tools/ ... (Leave 4GB for background processes)

10-17-2014, 03:06 AM	#7
GenoMax Senior Member Location: East Coast USA Join Date: Feb 2008 Posts: 7,092	Intel had presented some work they have done with NGS/GATK but I am not sure if the optimized code is available: http://bioinformatics.gatech.edu/sit...Processing.pdf

10-17-2014, 05:36 AM	#8
Baseless Member Location: Germany Join Date: Feb 2010 Posts: 32	On GATK it is possible on many tools to use the -nt or -nct options für using multiple threads. However, my advice is to go through every step with a benchmarking dataset to find where you get the best data per time ratio and maybe rather analyze multiple datasets at once than wase time by using more cores than it is beneficial for each step.