Tophat2 job fails with an out of memory error - how to restart?

[chris]

I've had a long running tophat2 end with an error after 9 nine days:

...
[2012-06-07 12:57:34] Joining segment hits
[2012-06-07 17:58:10] Reporting output tracks
[FAILED]
Error: [Errno 12] Cannot allocate memory
Found 280164 junctions from happy spliced reads

[FAILED]
Error: [Errno 2] No such file or directory: './tophat_out/tmp/accepted_hits7_sorted.bam'
Found 280164 junctions from happy spliced reads

Is there anyway to restart it at the point of failure on a machine with more memory rather than waiting another 9 nine days...?

[EGrassi]

I've managed to do so...you should check in your logs (run.log) the command that failed and re-launch it on a machine with more memory. In my case it worked and I only needed to launch by hand two or three commands.
You should try to understand which commands you still need to execute by examining the python wrapper for tophat and/or relaunching it on smaller fastq files (but with the same command line) and examining the logs. If it stopped at reporting output tracks it is similar to my situation: you should relaunch tophat-reports and samtools sort (in this way you will get an accepted_hits.bam but no unmapped.bam, I did not generate that file because I didn't need it...)

Be aware that this suggestion is not entirely "safe" as long as one can 'rebuild' the commands in the wrong way and skip eventual portions of the job done by the python...the developer suggested me to split my fastq files and run tophat on them separately to overcome memory limits.

[chris]

OK. It looks like it died trying to sort eight BAM files at the same time and then merging them into one. I'll try and run those steps separately and see how that goes...

Grazie mille