Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Memory issue with velvetg

    I'm new to de-novo assembly and hope to get some help with my memory issues. Here is what I'm trying to do:

    I want to assemble RNAseq data using velvet/oases. I have one HiSeq2000 lane of 100bp PE reads (approx. 170 M read pairs). This dataset has been randomly splitted into 10%, 20%, 30%....90% subsets. A colleague of mine managed to assemble the 100% dataset with approx. 104 GB RAM. I'm supposed to run the assemblies for the subsets. I managed to run the 10% and 20% samples through velvetg, velveth and oases without problems. The 30% sample i currently in the oases step. However, the 40% subset results in the following error messages:

    ------
    Exited with exit code 1.

    Resource usage summary:

    CPU time : 27907.75 sec.
    Max Memory : 55595 MB
    Max Swap : 92723 MB

    Max Processes : 4
    Max Threads : 52

    The output (if any) follows:

    velvetg: Can't malloc 540 ShortReadMarkers totalling 10800 bytes: Cannot allocate memory
    ------

    or:

    ------
    Exited with exit code 1.

    Resource usage summary:

    CPU time : 31115.64 sec.
    Max Memory : 40391 MB
    Max Swap : 42936 MB

    Max Processes : 4
    Max Threads : 52

    The output (if any) follows:

    velvetg: Can't malloc 267 ShortReadMarkers totalling 5340 bytes: Cannot allocate memory
    -------

    Velveth was run successfully with: ./velveth /Dir 27 -fastq -shortPaired /Dir/X.fastq -short /Dir/Y.fastq
    Velvetg settings are: ./velvetg /Dir -ins_length 300 -min_pair_count 2 -read_trkg yes -unused_reads yes

    In our AMD-Magny-Cours-Cluster (MEGWARE) Cluster I have 128 GB memory which should be enough (see above). The guys who are responsible for the core facility think that the problem might be that too many small portions of data are produced and that although there is enough memory in the system, it cannot deal with too many small packages (I hope this very unscientific description is clear enough). They suggested the incorporation of Boost (C++) Libraries?
    Additional information: The data is quality filtered and trimmed. The short reads are the singletons after trimming/filtering.

    I would appreciate any information on this issue.
    Thanks a lot in advance!

  • #2
    hi all,

    I did encounter the same issues as this post

    21686.720139] 19780000 / 35211509 nodes visited
    [21693.361691] 19790000 / 35211509 nodes visited

    velvetg: Can't malloc 254367 ShortReadMarkers totalling 2543670 bytes: Cannot allocate memory

    -----


    CPU time : 05:54:30
    Wall time : 06:01:47
    Mem usage : 93610144kb
    CPU usage : 98%

    I did allocate 256gb ram for this job, so there are plenty of memory to spare. But velvet just die unexpectedly..

    Any suggestion ?

    Comment


    • #3
      How many reads do you have and how long are the reads?

      There are a few things you could try that would help to use less memory.

      What parameters has velvet been compiled with?

      The parameters CATEGORIES and MAXKMERLENGTH affect the amount of memory velvet uses. If velvet has been compiled for more categories than you are using or a longer kmer length than you need, then reducing these values would reduce memory usage.

      Comment


      • #4
        Originally posted by mastal View Post
        How many reads do you have and how long are the reads?

        There are a few things you could try that would help to use less memory.

        What parameters has velvet been compiled with?

        The parameters CATEGORIES and MAXKMERLENGTH affect the amount of memory velvet uses. If velvet has been compiled for more categories than you are using or a longer kmer length than you need, then reducing these values would reduce memory usage.
        Hi,

        That was an Ion Proton reads, with read length ranging from 30-300 (after quality trimmed). I got around 25 million read per lib, 6 lib total.

        MAXKMERLENGTH is set at 70, CATEGORIES is 2

        Already read some of the opinions online and they suggest that data is so big so that even if the allocated memory is still available, the process cannot be continued.

        Comment

        Latest Articles

        Collapse

        • seqadmin
          Strategies for Sequencing Challenging Samples
          by seqadmin


          Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
          03-22-2024, 06:39 AM
        • seqadmin
          Techniques and Challenges in Conservation Genomics
          by seqadmin



          The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

          Avian Conservation
          Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
          03-08-2024, 10:41 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, Yesterday, 06:37 PM
        0 responses
        8 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, Yesterday, 06:07 PM
        0 responses
        8 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 03-22-2024, 10:03 AM
        0 responses
        49 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 03-21-2024, 07:32 AM
        0 responses
        67 views
        0 likes
        Last Post seqadmin  
        Working...
        X