SEQanswers

Go Back   SEQanswers > Sequencing Technologies/Companies > Pacific Biosciences



Similar Threads
Thread Thread Starter Forum Replies Last Post
Trinity RAM issue starbug16 Bioinformatics 8 02-12-2014 10:13 AM
Transcriptome assembly using 12GB RAM? ckidner Bioinformatics 7 09-14-2012 04:18 AM
Consumer RAM for NGS analysis JamesH Bioinformatics 5 02-14-2012 08:02 PM
Too many short reads and too little RAM? samanta General 5 09-22-2011 06:48 AM
assembly and RAM aurelielaugraud Bioinformatics 0 09-23-2009 07:02 AM

Reply
 
Thread Tools
Old 06-04-2014, 03:25 PM   #1
skerker
Member
 
Location: CA

Join Date: Nov 2009
Posts: 10
Default overlapInCore using up all RAM

I have been running RS_HGAP_Assembly.2 in SMRT portal and the job has been failing at the overlapInCore step. I was able to solve the problem by changing the nproc from 64 down to 24 so it would not use up all the available RAM on our system.

Each overlapInCore job was trying to use ~2.5GB of RAM, and the system only has 128GB. With 63 processes trying to run, the job failed. In order to optimize performance of the CA step - how do I determine how much RAM is used by each overlapInCore process?

For example, should I increase ovlThreads, while decreasing ovlConcurrency. I have 64 cores and 128GB RAM, so I'm trying to use as much available computing power as possible.

If anyone can provide insight on this, I'd appreciate it!

Best,
Jeff

From my runCA.spec file:


ovlThreads=integer (default=2)
The number of compute threads to use per overlap job.

ovlConcurrency=integer (default=1)
When not using SGE, the number of concurrent overlap jobs to run at the same time.

My current job is running with


ovlThreads = 1

ovlConcurrency = 24
skerker is offline   Reply With Quote
Old 06-05-2014, 02:27 PM   #2
rhall
Senior Member
 
Location: San Francisco

Join Date: Aug 2012
Posts: 322
Default

Optimizing the overlapper isn't particularly straightforward, the best explanation I have found is in the CA documentation:
http://sourceforge.net/apps/mediawik...hp?title=RunCA
in the "OVL Overlapper" section.
rhall is offline   Reply With Quote
Old 06-05-2014, 06:50 PM   #3
skerker
Member
 
Location: CA

Join Date: Nov 2009
Posts: 10
Default

Thanks for the reply. I haven't found much other information either. I guess I'll just have to play around with ovlThreads=integer (default=2) and ovlConcurrency=integer (default=1) and see what happens.

If threads = 2 and Concurrency = 12 is this the same as threads = 1 and concurrency = 24? That is the question I really was getting at.

Maybe it has something to do with how many threads can be processed on each CPU /core. All of this gets into the details of our actual hardware, which is beyond my expertise.

Jeff
skerker is offline   Reply With Quote
Old 06-06-2014, 08:16 AM   #4
rhall
Senior Member
 
Location: San Francisco

Join Date: Aug 2012
Posts: 322
Default

I believe, threads 1 and concurrency 24 will use approximately double the memory of threads 2 concurrency 12, if all other parameters are the same. As each concurrent job will allocate its own memory. Really optimizing the OVL requires altering the hash parameters, but that is data and system specific, and will likely involve some trial and error.
rhall is offline   Reply With Quote
Old 06-06-2014, 08:18 AM   #5
skerker
Member
 
Location: CA

Join Date: Nov 2009
Posts: 10
Default

Thank you - I appreciate the help and advice.

Jeff
skerker is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 02:56 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2021, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO