Tophat memory usage during "Searching for junctions via segment mapping"

[biznatch]

I'm using Tophat to align ~33 million 100 bp paired end reads to the mouse genome using -p 8 and and when it gets about 1 hour into "Searching for junctions via segment mapping" memory usage jumps from a few GB to almost 20 GB which exceeds the 16GB on my computer and starts using the swap file. I'm also using the "butterfly-search" which the Tophat manual says will slow slow down the alignment but doesn't say it should use more memory. Is this amount of memory usage normal or did something go wrong?

[robinpsmith]

I'm running Tophat2 on two 389MB paired end reads files, on 8 cores with 2.5G vmem per core (=20GB total) in SGE, using 16 threads.

The alignments go swimmingly (1 hour), then it takes ~19.7G of memory and 5+ hours to run the "Searching for junctions via segment mapping" step (my understanding is that the multiple threads are not used here). Then it makes it to "Mapping left_kept_reads_seg1 against segment_juncs with Bowtie2 (1/4)", but gets booted from the queue, probably due to memory usage.

If I run it with 4 cores, 16 threads, it gets booted at the "Searching for junctions via segment mapping" step (4 cores = only 10GB of memory).

I'm willing to go up to 10-12 cores, but looking for advice first. Any you care to provide would be greatly appreciated.

Thanks,

Robin

[yingzhang]

I am test driving TopHat 2.0 on a linux cluster.

I prepared 4 sample sets, which contain 13 million to 18 million 50bp paired reads. Tophat was called to process them one by one. I initialized the program with 8GB memory and 8 threads.

The first dataset was processed successfully using almost 2 hr. However, when the second dataset reached the step of "Building Bowtie index from genes.fa", the memory usage went above 18GB, so the system admin terminated my job.

So I re-started my job to only run TopHat 2.0 on dataset 2 using the same system setting. The job finished flawlessly.

I am wondering whether TopHat 2.0 has some memory issues.

[robinpsmith]

I found that removing the --fusion-search option substantially reduced memory usage. Not sure if this helps, but good luck.

[yingzhang]

I didn't use the option "--fusion-search". Based on the manual, I think the option is turned off by default.

[ramma]

Did anyone find a solution to this?

I'm mapping about 30 million 50bp reads to the human genome and during the "searching for junctions via segment mapping" step my memory usage sky rockets. About 2-4GB of memory is used before, then 24-25GB of memory is used and the run finally fails at the "Joining segment hits" step by exceeding my 32GB of memory. This is with the latest tophat2 and bowtie2.

Thanks for any help.

[robinpsmith]

Are you just using the vanilla options, or getting complicated? My tophat2 command that worked for is:

tophat2 -p 8 -m 2 -r 69 --mate-std-dev 200 -G genes.gtf -o ./out hg19 sample1_1.gz sample1_2.gz

But it does use a lot of memory - I think this one maxed at 16GB.

[ramma]

I'm doing mostly default options. My tophat command looks something like this:

tophat2 -p 10 --coverage-search --microexon-search -o /path/to/output /path/to/input1 /path/to/input2

[pettervikman]

Any more info regarding this? I just realised that my first 14 jobs that use --coverage-search --microexon-search have gone into junction search and they have started gobbling up memory (16-20Gb per sample). If they arnt finished when the next batch of 14 or even worse third batch of 14 comes to the same position I'm not certain that the servers got enough memory...

[ramma]

Originally Posted by pettervikman

Any more info regarding this? I just realised that my first 14 jobs that use --coverage-search --microexon-search have gone into junction search and they have started gobbling up memory (16-20Gb per sample). If they arnt finished when the next batch of 14 or even worse third batch of 14 comes to the same position I'm not certain that the servers got enough memory...

I've stopped using --coverage-search and --microexon-search on my personal server (32GB ram) as more than a single job at a time uses all the memory and kills itself. With those options disabled I've been able to run about 4-6 jobs at a time with no memory issues.