Hi, I have a set of 300 million 50-bp long SOLiD reads, and I would like to use BFAST to map them to a reference genome. I've successfully created the color-space reference genome .brfg file, and the 10 color-space reference genome indexes recommended in the BFAST book.
When trying 'bfast match -f reference_genome -A 1 -r reads.fastq > matches.bmf' in a 32 GB RAM machine, I got the memory corruption error copied below. Following BFAST book recommendations, I splitted the global fastq file in 10 filse with 30 million reads each. Since they also produced the same memory corruption error, I further splitted the reads in files with 3 million reads each. First three files were run successfully, but fourth file gave the same memory corruption error again. Then I splitted this file in 3 files with 1 million reads each. Again, first run was fine, but second run gave the same memory corruption error.
The strange thing is that running the fifth 3 million reads file (which is a little bigger than fourth one) did not give the memory corruption error. So, I do not think that I need more RAM because some 3 million reads could be processed, whereas some 1 million could not.
Any idea?
Files size are 33 GB for the 300 million reads fastq file, and 3.6 GB for each of 10 indexes
(...normal output here...)
Reading reads.fastq into temp files.
*** glibc detected *** bfast: malloc(): memory corruption: 0x000000000065cf30 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f8ad88d3a14]
/lib/libc.so.6(__libc_malloc+0x90)[0x7f8ad88d5360]
/lib/libc.so.6(vasprintf+0x3e)[0x7f8ad88ca66e]
/lib/libc.so.6(asprintf+0x88)[0x7f8ad88ad098]
/lib/libc.so.6(__assert_fail+0xb8)[0x7f8ad888a2a8]
bfast[0x41770d]
bfast[0x418927]
bfast[0x423b3b]
bfast[0x4259c5]
bfast[0x42b2d2]
/lib/libc.so.6(__libc_start_main+0xf4)[0x7f8ad887d1c4]
bfast[0x401ce9]
======= Memory map: ========
00400000-0044d000 r-xp 00000000 08:03 2711591 /usr/local/bin/bfast
0064d000-0064f000 rw-p 0004d000 08:03 2711591 /usr/local/bin/bfast
0064f000-010d8000 rw-p 0064f000 00:00 0 [heap]
7f8abc000000-7f8abc021000 rw-p 7f8abc000000 00:00 0
7f8abc021000-7f8ac0000000 ---p 7f8abc021000 00:00 0
7f8ac1c0c000-7f8ac1c19000 r-xp 00000000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1c19000-7f8ac1e19000 ---p 0000d000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1e19000-7f8ac1e1a000 rw-p 0000d000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1e1a000-7f8ad885f000 rw-p 7f8ac1e1a000 00:00 0
7f8ad885f000-7f8ad89b7000 r-xp 00000000 08:03 647441 /lib/libc-2.7.so
7f8ad89b7000-7f8ad8bb7000 ---p 00158000 08:03 647441 /lib/libc-2.7.so
7f8ad8bb7000-7f8ad8bba000 r--p 00158000 08:03 647441 /lib/libc-2.7.so
7f8ad8bba000-7f8ad8bbc000 rw-p 0015b000 08:03 647441 /lib/libc-2.7.so
7f8ad8bbc000-7f8ad8bc1000 rw-p 7f8ad8bbc000 00:00 0
7f8ad8bc1000-7f8ad8bd7000 r-xp 00000000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8bd7000-7f8ad8dd7000 ---p 00016000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8dd7000-7f8ad8dd9000 rw-p 00016000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8dd9000-7f8ad8ddd000 rw-p 7f8ad8dd9000 00:00 0
7f8ad8ddd000-7f8ad8dec000 r-xp 00000000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8dec000-7f8ad8feb000 ---p 0000f000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8feb000-7f8ad8fed000 rw-p 0000e000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8fed000-7f8ad9003000 r-xp 00000000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9003000-7f8ad9203000 ---p 00016000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9203000-7f8ad9204000 rw-p 00016000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9204000-7f8ad9284000 r-xp 00000000 08:03 647446 /lib/libm-2.7.so
7f8ad9284000-7f8ad9483000 ---p 00080000 08:03 647446 /lib/libm-2.7.so
7f8ad9483000-7f8ad9485000 rw-p 0007f000 08:03 647446 /lib/libm-2.7.so
7f8ad9485000-7f8ad94a2000 r-xp 00000000 08:03 647435 /lib/ld-2.7.so
7f8ad9543000-7f8ad9694000 rw-p 7f8ad9543000 00:00 0
7f8ad969d000-7f8ad96a2000 rw-p 7f8ad969d000 00:00 0
7f8ad96a2000-7f8ad96a4000 rw-p 0001d000 08:03 647435 /lib/ld-2.7.so
7fff956b3000-7fff956c8000 rw-p 7ffffffea000 00:00 0 [stack]
7fff957fe000-7fff95800000 r-xp 7fff957fe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
When trying 'bfast match -f reference_genome -A 1 -r reads.fastq > matches.bmf' in a 32 GB RAM machine, I got the memory corruption error copied below. Following BFAST book recommendations, I splitted the global fastq file in 10 filse with 30 million reads each. Since they also produced the same memory corruption error, I further splitted the reads in files with 3 million reads each. First three files were run successfully, but fourth file gave the same memory corruption error again. Then I splitted this file in 3 files with 1 million reads each. Again, first run was fine, but second run gave the same memory corruption error.
The strange thing is that running the fifth 3 million reads file (which is a little bigger than fourth one) did not give the memory corruption error. So, I do not think that I need more RAM because some 3 million reads could be processed, whereas some 1 million could not.
Any idea?
Files size are 33 GB for the 300 million reads fastq file, and 3.6 GB for each of 10 indexes
(...normal output here...)
Reading reads.fastq into temp files.
*** glibc detected *** bfast: malloc(): memory corruption: 0x000000000065cf30 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f8ad88d3a14]
/lib/libc.so.6(__libc_malloc+0x90)[0x7f8ad88d5360]
/lib/libc.so.6(vasprintf+0x3e)[0x7f8ad88ca66e]
/lib/libc.so.6(asprintf+0x88)[0x7f8ad88ad098]
/lib/libc.so.6(__assert_fail+0xb8)[0x7f8ad888a2a8]
bfast[0x41770d]
bfast[0x418927]
bfast[0x423b3b]
bfast[0x4259c5]
bfast[0x42b2d2]
/lib/libc.so.6(__libc_start_main+0xf4)[0x7f8ad887d1c4]
bfast[0x401ce9]
======= Memory map: ========
00400000-0044d000 r-xp 00000000 08:03 2711591 /usr/local/bin/bfast
0064d000-0064f000 rw-p 0004d000 08:03 2711591 /usr/local/bin/bfast
0064f000-010d8000 rw-p 0064f000 00:00 0 [heap]
7f8abc000000-7f8abc021000 rw-p 7f8abc000000 00:00 0
7f8abc021000-7f8ac0000000 ---p 7f8abc021000 00:00 0
7f8ac1c0c000-7f8ac1c19000 r-xp 00000000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1c19000-7f8ac1e19000 ---p 0000d000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1e19000-7f8ac1e1a000 rw-p 0000d000 08:03 647184 /lib/libgcc_s.so.1
7f8ac1e1a000-7f8ad885f000 rw-p 7f8ac1e1a000 00:00 0
7f8ad885f000-7f8ad89b7000 r-xp 00000000 08:03 647441 /lib/libc-2.7.so
7f8ad89b7000-7f8ad8bb7000 ---p 00158000 08:03 647441 /lib/libc-2.7.so
7f8ad8bb7000-7f8ad8bba000 r--p 00158000 08:03 647441 /lib/libc-2.7.so
7f8ad8bba000-7f8ad8bbc000 rw-p 0015b000 08:03 647441 /lib/libc-2.7.so
7f8ad8bbc000-7f8ad8bc1000 rw-p 7f8ad8bbc000 00:00 0
7f8ad8bc1000-7f8ad8bd7000 r-xp 00000000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8bd7000-7f8ad8dd7000 ---p 00016000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8dd7000-7f8ad8dd9000 rw-p 00016000 08:03 647459 /lib/libpthread-2.7.so
7f8ad8dd9000-7f8ad8ddd000 rw-p 7f8ad8dd9000 00:00 0
7f8ad8ddd000-7f8ad8dec000 r-xp 00000000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8dec000-7f8ad8feb000 ---p 0000f000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8feb000-7f8ad8fed000 rw-p 0000e000 08:03 647294 /lib/libbz2.so.1.0.4
7f8ad8fed000-7f8ad9003000 r-xp 00000000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9003000-7f8ad9203000 ---p 00016000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9203000-7f8ad9204000 rw-p 00016000 08:03 4089806 /usr/lib/libz.so.1.2.3.3
7f8ad9204000-7f8ad9284000 r-xp 00000000 08:03 647446 /lib/libm-2.7.so
7f8ad9284000-7f8ad9483000 ---p 00080000 08:03 647446 /lib/libm-2.7.so
7f8ad9483000-7f8ad9485000 rw-p 0007f000 08:03 647446 /lib/libm-2.7.so
7f8ad9485000-7f8ad94a2000 r-xp 00000000 08:03 647435 /lib/ld-2.7.so
7f8ad9543000-7f8ad9694000 rw-p 7f8ad9543000 00:00 0
7f8ad969d000-7f8ad96a2000 rw-p 7f8ad969d000 00:00 0
7f8ad96a2000-7f8ad96a4000 rw-p 0001d000 08:03 647435 /lib/ld-2.7.so
7fff956b3000-7fff956c8000 rw-p 7ffffffea000 00:00 0 [stack]
7fff957fe000-7fff95800000 r-xp 7fff957fe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Comment