Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • overlapInCore using up all RAM

    I have been running RS_HGAP_Assembly.2 in SMRT portal and the job has been failing at the overlapInCore step. I was able to solve the problem by changing the nproc from 64 down to 24 so it would not use up all the available RAM on our system.

    Each overlapInCore job was trying to use ~2.5GB of RAM, and the system only has 128GB. With 63 processes trying to run, the job failed. In order to optimize performance of the CA step - how do I determine how much RAM is used by each overlapInCore process?

    For example, should I increase ovlThreads, while decreasing ovlConcurrency. I have 64 cores and 128GB RAM, so I'm trying to use as much available computing power as possible.

    If anyone can provide insight on this, I'd appreciate it!

    Best,
    Jeff

    From my runCA.spec file:


    ovlThreads=integer (default=2)
    The number of compute threads to use per overlap job.

    ovlConcurrency=integer (default=1)
    When not using SGE, the number of concurrent overlap jobs to run at the same time.

    My current job is running with


    ovlThreads = 1

    ovlConcurrency = 24

  • #2
    Optimizing the overlapper isn't particularly straightforward, the best explanation I have found is in the CA documentation:
    http://sourceforge.net/apps/mediawik...hp?title=RunCA
    in the "OVL Overlapper" section.

    Comment


    • #3
      Thanks for the reply. I haven't found much other information either. I guess I'll just have to play around with ovlThreads=integer (default=2) and ovlConcurrency=integer (default=1) and see what happens.

      If threads = 2 and Concurrency = 12 is this the same as threads = 1 and concurrency = 24? That is the question I really was getting at.

      Maybe it has something to do with how many threads can be processed on each CPU /core. All of this gets into the details of our actual hardware, which is beyond my expertise.

      Jeff

      Comment


      • #4
        I believe, threads 1 and concurrency 24 will use approximately double the memory of threads 2 concurrency 12, if all other parameters are the same. As each concurrent job will allocate its own memory. Really optimizing the OVL requires altering the hash parameters, but that is data and system specific, and will likely involve some trial and error.

        Comment


        • #5
          Thank you - I appreciate the help and advice.

          Jeff

          Comment

          Latest Articles

          Collapse

          • seqadmin
            Strategies for Sequencing Challenging Samples
            by seqadmin


            Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
            03-22-2024, 06:39 AM
          • seqadmin
            Techniques and Challenges in Conservation Genomics
            by seqadmin



            The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

            Avian Conservation
            Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
            03-08-2024, 10:41 AM

          ad_right_rmr

          Collapse

          News

          Collapse

          Topics Statistics Last Post
          Started by seqadmin, Yesterday, 06:37 PM
          0 responses
          8 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, Yesterday, 06:07 PM
          0 responses
          8 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-22-2024, 10:03 AM
          0 responses
          49 views
          0 likes
          Last Post seqadmin  
          Started by seqadmin, 03-21-2024, 07:32 AM
          0 responses
          67 views
          0 likes
          Last Post seqadmin  
          Working...
          X