View Single Post
Old 12-16-2016, 07:12 PM   #17
chiayi
Member
 
Location: New York

Join Date: Dec 2016
Posts: 22
Default

Thank you so much for the thorough explanation. I tried a couple things and please find the reports as follows:

Quote:
Originally Posted by Brian Bushnell View Post
Personally, I consider this to be a major bug in the job schedulers that have this behavior. Also, not allowing programs to over-commit virtual memory (meaning, use more virtual memory than is physically present) is generally a very bad idea. Virtual memory is free, after all. What job scheduler are you using? And do you know what your cluster's policy is for over-comitting virtual memory?
I'm not sure about the answers to these two questions. I will need to ask around and get back to you.

Quote:
Anyway, please try with requesting 48GB and using the -Xmx12g flag (or alternately requesting 16GB and using -Xmx4g) and let me know if that resolves the problem.
Still ran out of memory with 48GB r'q and -Xmx12g tag:

HTML Code:
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at stream.FASTQ.makeId(FASTQ.java:567)
    at stream.FASTQ.quadToRead(FASTQ.java:785)
    at stream.FASTQ.toReadList(FASTQ.java:710)
    at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
    at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
    at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
    at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)

This program ran out of memory.
Try increasing the -Xmx flag and using tool-specific memory-related parameters.
Quote:
So if you need to set -Xmx manually because the memory autodetection does not work (in which case, I'd like to hear the details about what the program does when you don't define -Xmx, because I want to make it as easy to use as possible) then please allow some overhead.
When I requested 16GB and did not specify -Xmx, the program matched the requested memory,
java -ea -Xmx17720m -Xms17720m -cp
chiayi is offline   Reply With Quote