Cufflinks memory usage

oscarluoinau · 11-08-2011, 11:51 PM

Hi all,
I am having trouble with running Cufflinks on PE RNA-Seq libraries generated from HiSeq machine. I have used Tophat to successfully mapped those PE reads (about 160 millions reads), which gave me a BAM file about 6GB for each library. Then I fed the BAM file to Cufflinks running with 20 cores. Now the problem is it seems Cufflinks is taking more than 120GB ram, and is taking very long (about a week) to run one library. Have any of you had similar experience? Am I doing something wrong? Any suggestions? Thanks!

Nicolas · 11-09-2011, 08:05 AM

Which reference annotations are you using? I had similar experience with gencode annotations (>2.5 millions annotations). I then switched to RefSeq annotations and the Cufflinks runs are now much shorter. Of course, it's quantify much less potential transcripts, but for most applications, that can be sufficient.

jbrwn · 11-09-2011, 02:30 PM

in addition to posting your reference, you may want to post which options you're utilizing in cufflinks.

oscarluoinau · 11-09-2011, 06:05 PM

Right. I am also using the Gencode annotation. I think I will experiment with other annotation file to see how it goes. The options I am utilizing in Cufflinks are simply specifying the reads come from PE reads (i.e. --fr-unstranded).

kjlee · 12-08-2012, 06:40 PM

oscar,

did you ever get an answer or a work around? I am running cufflinks on a similar sized bam file and am also running out of memory.

cheers,

oscarluoinau · 12-10-2012, 06:21 AM

I couldn't get the job done until upgraded to the latest version of Cufflinks which seems to use less memory. Good luck!

kjlee · 12-10-2012, 06:27 AM

Hey Oscar,

I am running the newest version of Cufflinks (I literally downloaded it this week). I got Cufflinks to run on a small sub-set of reads (about 4 million paired end reads, 100nt with ~200nt inner gap). But the whole data-set is ~50X bigger. How many reads did you use? And what (approximately) was the memory usage for your file size (RAM per GB of the bam/sam file)?

Cheers,

oscarluoinau · 12-10-2012, 06:33 AM

Hi,
I don't remember the exact numbers as it was about a year ago. One thing I do remember is I was lucky enough to utilize a machine with 1TB RAM, and I used about 500GB for about 160 million reads. I hope this helps. Good luck!

kjlee · 12-10-2012, 06:50 AM

We only have one compute node with that much memory on our cluster and I didn't want to usurp it if I didn't have to. But I guess that's what the resources are for. Thanks Oscar. (Also, about how long did it take to run?)

Cheers,

oscarluoinau · 12-10-2012, 07:55 AM

Expect it to run longer than a week.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
segment_juncs memory usage while running Tophat	genec	Bioinformatics	1	11-22-2011 08:09 AM
bwa mt branch extreme memory usage?	Kotoro	Bioinformatics	4	11-20-2011 01:17 AM
Memory usage	Seta	General	2	04-05-2011 11:44 AM
Memory Usage in Newbler 2.3	smg283	454 Pyrosequencing	5	11-09-2010 08:46 AM
SHRiMP Memory Usage	DNAjunk	Bioinformatics	5	08-05-2009 03:07 PM

11-08-2011, 11:51 PM	#1
oscarluoinau Junior Member Location: Farmington, CT Join Date: Nov 2011 Posts: 6	Cufflinks memory usage Hi all, I am having trouble with running Cufflinks on PE RNA-Seq libraries generated from HiSeq machine. I have used Tophat to successfully mapped those PE reads (about 160 millions reads), which gave me a BAM file about 6GB for each library. Then I fed the BAM file to Cufflinks running with 20 cores. Now the problem is it seems Cufflinks is taking more than 120GB ram, and is taking very long (about a week) to run one library. Have any of you had similar experience? Am I doing something wrong? Any suggestions? Thanks!

11-09-2011, 08:05 AM	#2
Nicolas Member Location: new york city Join Date: Apr 2009 Posts: 40	Which reference annotations are you using? I had similar experience with gencode annotations (>2.5 millions annotations). I then switched to RefSeq annotations and the Cufflinks runs are now much shorter. Of course, it's quantify much less potential transcripts, but for most applications, that can be sufficient.

11-09-2011, 02:30 PM	#3
jbrwn Member Location: Denver, CO Join Date: Mar 2011 Posts: 37	in addition to posting your reference, you may want to post which options you're utilizing in cufflinks.

11-09-2011, 06:05 PM	#4
oscarluoinau Junior Member Location: Farmington, CT Join Date: Nov 2011 Posts: 6	Right. I am also using the Gencode annotation. I think I will experiment with other annotation file to see how it goes. The options I am utilizing in Cufflinks are simply specifying the reads come from PE reads (i.e. --fr-unstranded).

12-08-2012, 06:40 PM	#5
kjlee Member Location: Atlanta, GA Join Date: Jun 2011 Posts: 12	oscar, did you ever get an answer or a work around? I am running cufflinks on a similar sized bam file and am also running out of memory. cheers,

12-10-2012, 06:21 AM	#6
oscarluoinau Junior Member Location: Farmington, CT Join Date: Nov 2011 Posts: 6	I couldn't get the job done until upgraded to the latest version of Cufflinks which seems to use less memory. Good luck!

12-10-2012, 06:27 AM	#7
kjlee Member Location: Atlanta, GA Join Date: Jun 2011 Posts: 12	Hey Oscar, I am running the newest version of Cufflinks (I literally downloaded it this week). I got Cufflinks to run on a small sub-set of reads (about 4 million paired end reads, 100nt with ~200nt inner gap). But the whole data-set is ~50X bigger. How many reads did you use? And what (approximately) was the memory usage for your file size (RAM per GB of the bam/sam file)? Cheers,

12-10-2012, 06:33 AM	#8
oscarluoinau Junior Member Location: Farmington, CT Join Date: Nov 2011 Posts: 6	Hi, I don't remember the exact numbers as it was about a year ago. One thing I do remember is I was lucky enough to utilize a machine with 1TB RAM, and I used about 500GB for about 160 million reads. I hope this helps. Good luck!

12-10-2012, 06:50 AM	#9
kjlee Member Location: Atlanta, GA Join Date: Jun 2011 Posts: 12	We only have one compute node with that much memory on our cluster and I didn't want to usurp it if I didn't have to. But I guess that's what the resources are for. Thanks Oscar. (Also, about how long did it take to run?) Cheers,

12-10-2012, 07:55 AM	#10
oscarluoinau Junior Member Location: Farmington, CT Join Date: Nov 2011 Posts: 6	Expect it to run longer than a week.