SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
crossmatch speed? ewilbanks Bioinformatics 0 07-14-2014 11:44 AM
how to speed up minimus2 jjjscuedu Bioinformatics 2 04-12-2013 10:47 AM
Cuffdiff: Importance Sampling and Variance Calculation jk1124 RNA Sequencing 0 10-25-2012 02:43 PM
how can i speed up bwa? yujiro Bioinformatics 21 04-20-2011 07:12 PM

Reply
 
Thread Tools
Old 11-28-2017, 04:35 AM   #1
capricy
Senior Member
 
Location: 63130

Join Date: Apr 2012
Posts: 120
Default speed up the cuffdiff calculation

Dear Community,

I have 12 sets of RNAseq data (4 groups) and mapped them using STAR. The resultant bam files ranged from 4G to 13 G. I then run cuffdiff on those bam files to get the FPKMs and differential genes. The process has been on-going for over a day and the log file still stopped at " Inspecting maps and determining fragment length distributions". I wonder if the cuffdiff has limit about the sample size? Is it normal this slow?

Thanks a lot for any inputs!

C.
capricy is offline   Reply With Quote
Old 11-28-2017, 08:54 PM   #2
neavemj
Member
 
Location: MA, USA

Join Date: Feb 2014
Posts: 40
Default

Hi Capricy,

I recently used cuffdiff on 12 samples (bam sizes ~2Gb each) and it took 5-6 hours. This was using a server with 128Gb and 20 threads.

I'd say your process does seem a bit slow, although your bam files are larger than mine. How much memory / threads are you using? I wonder if you could use the 'top' command to check if the program is still actually running?

Cheers,

Matt.
neavemj is offline   Reply With Quote
Old 11-29-2017, 03:47 AM   #3
capricy
Senior Member
 
Location: 63130

Join Date: Apr 2012
Posts: 120
Default

Hi, Matt,

Thank you very much for reply.

I am running on HPC. I used 96G mem, 40 processor. They are still running..., and last night I started to see the output for file: var_model.info

I wonder if the uneven file size would be the issue.

Not sure how long it would take to actually finish.

C.

Last edited by capricy; 11-29-2017 at 03:55 AM.
capricy is offline   Reply With Quote
Old 11-29-2017, 01:59 PM   #4
neavemj
Member
 
Location: MA, USA

Join Date: Feb 2014
Posts: 40
Default

hmm, well at least it's not just hanging!

Not sure why it's taking that long. If you've given cuffdiff all the threads '-p 40', I'd have thought that would be plenty. Maybe someone else has a better idea?

Matt.
neavemj is offline   Reply With Quote
Old 11-30-2017, 10:37 AM   #5
Markiyan
Member
 
Location: Cambridge

Join Date: Sep 2010
Posts: 99
Lightbulb After 8 theads the speedup for cufflinks/diff is marginal...

From my experience the speed up of the cufflinks/cuffdiff is marginal after 8 threads...

In some cases the runtime with 32-48 threads may be way longer than with 8-16, esp on systems with 4+ CPU sockets due to bottlenecks caused by memory interconnects saturation/latencies.

Also make sure the system/program is using NUMA properly and cpu interleaving is not set in the BIOS setup.

For tophat/cufflinks I would run several jobs using 1-8 threads in parallel than one job at a time using 40 threads in series (provided enough ram is available).

PS: And be patient... - leave job running overnight/weekend/Christmas Holiday :-)
Markiyan is offline   Reply With Quote
Old 11-30-2017, 12:02 PM   #6
capricy
Senior Member
 
Location: 63130

Join Date: Apr 2012
Posts: 120
Default

Thank you very much for advice about bringing down the value for -p.

I will try that with larger memory.

Actually all my jobs are hanging at:

ChkbCpt1b
> Processing Locus chr15:100479569-100495239 [******************** ] 81%Methig1
Mettl7a2
Methig1
Mettl7a2
> Processing Locus chr15:100469033-100479252 [******************** ] 81%Methig1
Mettl7a2
Methig1
Mettl7a2
> Processing Locus chr15:103562759-103565081 [******************** ] 81%

I am working on mouse data. I use mm10 gtf as reference.

C.
capricy is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 08:58 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO