Sorting large files

[scami]

Hi there,

I used soap2 in order to align 45 Gig ilumina reads over an indexed reference. Now I should use soapsnp to get the consensus sequence and to call snp. The output of sopa2 should be sorted though! The problem is that if I have a 153 Gigb file (which represent only chromosome 1 !!). How can I sort such a huge file? I was trying the unix sort command, but will it use an amount of memory comparable to the size of the file to sort (I don't know whether I will be able to access a computer with such an amount of free memory)?

thanks for your help

[adamdeluca]

I have used the unix sort command for very large files. It keeps only a small buffer in memory. You can increase the buffer size with -S to improve performance. Test this for yourself by sorting a 1GB file and watching the memory use.

[scami]

Great news!

I'm gonna go for it

thanks

[Torst]

Originally Posted by scami

How can I sort such a huge file? I was trying the unix sort command, but will it use an amount of memory comparable to the size of the file to sort (I don't know whether I will be able to access a computer with such an amount of free memory)?

Unix "sort" uses a smart merge sort, and utilises temporary files on disk, so spare disk space is an issue. By default it uses /tmp to store these file (or $TMPDIR), but you can change this with the -T option.

As already mentioned, you can use the -S option to increase buffer sizes. You can use it like "-S 80%" to use 80% of available RAM, or "-S 2G" for 2 GB RAM.

For more information, read the texinfo page by typing "pinfo sort" or "info sort".