Trinity RAM issue

[starbug16]

Hi,

Having managed to whittle down my Illumina data to 31 million reads, I am trying to run Trinity with 16 CPUs and 16 gigabytes of RAM. Everything works fine until I get to Butterfly which seems to bog down the server so much so that I can't even connect in to check progress and end up having to kill the process.

Guessing that I don't have enough RAM. Does anyone have any suggestions on how to work around this???

Thanks in advance

[Birdman]

What is your Trinity command line?

[Wallysb01]

For butterfly you’re going to have to drop the CPUs and RAM usage down from defaults or from what you set in other parts of the trinity run. You can control that with:

--bflyHeapSpaceMax
--bflyCPU

You should probably just be running with two CPUs and 8GB of ram set as the bflyHeapSpaceMax

[starbug16]

Thanks, will definitely give that a try. Probably a how long is a piece of string question but...how long do you think it will take to complete?

[Birdman]

I really depends on the amount of reads you have. As an example, in my experience using the latest Trinity version, 1 billion reads took 6-7 days using 64 cores and 256G memory.

[Wallysb01]

with only 2 cores running and 31M reads, hopefully run time is not much more than 24 hours.

[starbug16]

OK, so running on 2 cores and it's run 0.8% of commands overnight. Guessing it may take a while Not sure there is a coffee break long enough for this run

[Wallysb01]

You could drop the RAM usage for each butterfly run more to get 4 cores going instead of 2, but you’ll risk having a lot of jobs to rerun.

Why are you working with only 16GB of RAM but 16 cores? Is this a machine with DDR2 memory? Can you upgrade that RAM? A modern desktop should be able to get you up to 32 GB, and a relatively cheap workstation (i.e. $5000) will give you 16 cores and 128GB of RAM. If you’re going to be doing this much it would be worth it to get something modern or finding a cluster with something better.

[starbug16]

Ah, just the usual academic institution tug of war between money needing to be spent to be able to do research and them wanting the work in already to justify it. This server was never supposed to be for assembly and they aren't necessarily willing to stump up the cash to add the RAM A month wait for their assembly should do the trick

Thanks for the advice guys, it's much appreciated it's at least running now, just going to take a wee while!