SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
BFAST malloc error CS Student General 1 01-11-2012 06:16 PM
BFAST index error kursuni Bioinformatics 1 09-27-2011 01:39 AM
Bfast localalign error dg.pooja General 1 01-31-2011 06:35 PM
bfast localalign error nimmi Bioinformatics 6 01-05-2011 09:40 AM
malloc error in cuffcompare genec Bioinformatics 0 04-06-2010 12:20 PM

Reply
 
Thread Tools
Old 02-01-2011, 11:47 PM   #1
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default BFAST malloc error under OS X

Hi

I have been using BFAST successfully for ABI alignment to hg18 for a while now. I have fresh installs of versions 0.6.1c, 0.6.3c and the latest 0.6.4e. Each version is make by

sh autogen.sh
./configure
make


Depending which I point PATH to I find that the first two work fine under Mac OS 10.5.8 on a lowly Duocore/4Gb machine, but the most recent version falls over when making a reference index from an exons files.

So

bfast index -f hg18all.fa -A 1 -n 2 -m 1111111111111111111111 -w 14 -t -i 1 -x exonFile.txt -T /tmp/

... runs fine for the first two versions, but with 0.6.4e gives a malloc error (terminal output pasted below). Interestingly, the 0.6.4e version does run successfully on my Snow Leopard laptop, suggesting some sensitivity to the background OS.

Best regards

David

=========


dk$ bfast index -f $hg18dir/hg18all.fa -A 1 -n 2 -m 1111111111111111111111 -w 14 -t -i 1 -x $exondir/pan73regions.txt -T /tmp/
************************************************************
Checking input parameters supplied by the user ...
Validating fastaFileName /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa.
Validating exonsFileName /Users/dk/Desktop/pan/documentation/pan73regions.txt.
Validating tmpDir path /tmp/.
Input arguments look good!
************************************************************
************************************************************
Printing Program Parameters:
programMode: [ExecuteProgram]
fastaFileName: /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa
space: [Color Space]
mask: 1111111111111111111111
depth: 0
hashWidth: 14
indexNumber: 1
repeatMasker: [Not Using]
startContig: 0
startPos: 0
endContig: 2147483647
endPos: 2147483647
exonsFileName: /Users/dk/Desktop/pan/documentation/pan73regions.txt
numThreads: 2
tmpDir: /tmp/
timing: [Using]
************************************************************
Reading in exons from /Users/dk/Desktop/pan/documentation/pan73regions.txt.
Read in 73 exons.
************************************************************
Reading in reference genome from /Users/dk/Desktop/pan/data/hg18Genome/hg18all.fa.cs.brg.
In total read 24 contigs for a total of 3080419480 bases
************************************************************
Creating the index...
************************************************************
Warning: startContig was less than zero.
Defaulting to contig=1 and position=1.
************************************************************
************************************************************
Warning: endContig was greater than the number of contigs in the reference genome.
Defaulting to reference genome's end contig=24 and position=57772954.
************************************************************
Currently on [contig,pos]:
[------24,---57772954]
Sorting by thread...
Merging sorts from threads...
Out of 1 required merges, currently on:
1
Merge complete.
Sorted.
bfast(20212,0xa00e8720) malloc: *** mmap(size=1073741824) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
************************************************************
In function "RGIndexCreateHash": Fatal Error[MallocMemory]. Variable/Value: index->starts.
Message: Could not allocate memory.
***** Exiting due to errors *****
**********************************************************
**
davidkip is offline   Reply With Quote
Old 02-01-2011, 11:54 PM   #2
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Looks like you ran out of memory, how much do you have in the failing and successful machines respectively?
nilshomer is offline   Reply With Quote
Old 02-02-2011, 12:04 AM   #3
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default

4Gb in failed machine, 8Gb in successful one. The index (when made) is reported as being 1.09Gb in size (~86Mb file). Watching the memory usage shows BFAST using around 1.3Gb of memory (and there still being some free) at the time of crash.

The exons file covers ~ 14.5Mb of the human genome.

Curiously, the 0.6.3c version on the 4Gb machine works fine with no memory issues. Indeed, I've been able to make some very large indexes in this environment (eg all of hg18, split into 16 subindexes).
davidkip is offline   Reply With Quote
Old 02-02-2011, 12:09 AM   #4
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

There is no difference in the indexing portion between the versions, so I am at a loss.
nilshomer is offline   Reply With Quote
Old 02-02-2011, 12:29 AM   #5
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default

Is fasta2brg identical as well (as the dependent earlier step)?

Anything I can do to help debug this?
davidkip is offline   Reply With Quote
Old 02-02-2011, 07:21 AM   #6
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default

Update:

Following on from your comment that the indexing is identical in the different versions, I made an index under an earlier version and then used that as the input to the rest of the workflow (now switched to the more update BFAST).

Result: bfast match falls over with a malloc error as well.

This sounds like a general compilation difference between the two versions?

After doing

sh autogen.sg && ./configure

on freshly unpacked tarballs of each version, I did a file difference to see the difference between the makefiles (see below). Apart from a few lines where the version numbers change, a removal of the solid2fastq.pl script in the current release, there are two main differences:

91c91
< CFLAGS = -Wall -g -O2 -pthread
---
> CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64
110c110
< LIBS = -lz -lm -lbz2
---
> LIBS = -lbz2 -lz -lz -lm


Does this give any clues as to what is going on?

All the best

David

===============



$ diff /Users/dk/Desktop/ngs/bfast-0.6.4e/Makefile /Users/dk/Desktop/ngs/bfast-0.6.3c/Makefile
83,87c83,87
< ACLOCAL = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run aclocal-1.10
< AMTAR = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run tar
< AUTOCONF = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run autoconf
< AUTOHEADER = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run autoheader
< AUTOMAKE = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run automake-1.10
---
> ACLOCAL = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run aclocal-1.10
> AMTAR = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run tar
> AUTOCONF = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run autoconf
> AUTOHEADER = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run autoheader
> AUTOMAKE = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run automake-1.10
91c91
< CFLAGS = -Wall -g -O2 -pthread
---
> CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64
110c110
< LIBS = -lz -lm -lbz2
---
> LIBS = -lbz2 -lz -lz -lm
112c112
< MAKEINFO = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.4e/config/missing --run makeinfo
---
> MAKEINFO = ${SHELL} /Users/dk/Desktop/ngs/bfast-0.6.3c/config/missing --run makeinfo
118c118
< PACKAGE_STRING = bfast 0.6.4e
---
> PACKAGE_STRING = bfast 0.6.3c
120c120
< PACKAGE_VERSION = 0.6.4e
---
> PACKAGE_VERSION = 0.6.3c
125,129c125,129
< VERSION = 0.6.4e
< abs_builddir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_top_builddir = /Users/dk/Desktop/ngs/bfast-0.6.4e
< abs_top_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.4e
---
> VERSION = 0.6.3c
> abs_builddir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_top_builddir = /Users/dk/Desktop/ngs/bfast-0.6.3c
> abs_top_srcdir = /Users/dk/Desktop/ngs/bfast-0.6.3c
152c152
< install_sh = $(SHELL) /Users/dk/Desktop/ngs/bfast-0.6.4e/config/install-sh
---
> install_sh = $(SHELL) /Users/dk/Desktop/ngs/bfast-0.6.3c/config/install-sh
173a174
> scripts/solid2fastq.pl \




davidkip is offline   Reply With Quote
Old 02-02-2011, 10:54 AM   #7
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Try setting the CFLAGS back to "CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64" and see what happens.
nilshomer is offline   Reply With Quote
Old 02-03-2011, 12:24 AM   #8
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default

OK, I've tried that now.

Fresh unpack of each tarball, ./configure in each, then edit the Makefile for the new version to have the addition -m64 etc tag.

No success, but I think I see why from the compilation output. It looks like the flag isn't passed to gcc just by altering this Makefile [I am a complete newbie w.r.t. C compilation BTW]. So the output for the older (working) version is:


$ make
make all-recursive
Making all in bfast
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64 -MT AlignedEnd.o -MD -MP -MF .deps/AlignedEnd.Tpo -c -o AlignedEnd.o AlignedEnd.c
mv -f .deps/AlignedEnd.Tpo .deps/AlignedEnd.Po
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64 -MT AlignedEntry.o -MD -MP -MF .deps/AlignedEntry.Tpo -c -o AlignedEntry.o AlignedEntry.c

...etc.

The output for the new version after editing the Makefile is

$ make
make all-recursive
Making all in bfast
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -MT AlignedEnd.o -MD -MP -MF .deps/AlignedEnd.Tpo -c -o AlignedEnd.o AlignedEnd.c
mv -f .deps/AlignedEnd.Tpo .deps/AlignedEnd.Po
gcc -DHAVE_CONFIG_H -I. -I.. -Wall -g -O2 -pthread -MT AlignedEntry.o -MD -MP -MF .deps/AlignedEntry.Tpo -c -o AlignedEntry.o AlignedEntry.c

...etc.

Does that help?
davidkip is offline   Reply With Quote
Old 02-03-2011, 03:36 PM   #9
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

In configure.ac, set "CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64". The run "sh autogen && ./configure".
nilshomer is offline   Reply With Quote
Old 02-04-2011, 02:00 AM   #10
davidkip
Junior Member
 
Location: Cardiff, UK

Join Date: Dec 2009
Posts: 6
Default

Success!

In the config.ac file for older versions there is this line:


extended_CFLAGS="-m64 -D_FILE_OFFSET_BITS=64";

but in the latest version it is commented out to this:

extended_CFLAGS="";# "-m64 -D_FILE_OFFSET_BITS=64";


Changing it to the earlier version of this line results in the m64 flags being passed to gcc, and the eventual build of the current release now works perfectly. Sorted!

Thanks for your help Nils.
davidkip is offline   Reply With Quote
Old 08-30-2011, 03:50 AM   #11
seeker
Member
 
Location: Zurich

Join Date: Jan 2011
Posts: 26
Default

I am having a similar error running bfast match. It will go through one index (of 10 indexes) fine and then give the same malloc error followed by a message saying that it could not allocate enough memory on the second index.

I edited the config.ac to:

extended_CFLAGS="-m64 -D_FILE_OFFSET_BITS=64";

which got me through the first couple indexes and then the same error. I then edited:

CFLAGS = -Wall -g -O2 -pthread -m64 -D_FILE_OFFSET_BITS=64

Which got me through the first four but then gave an error. Interestingly, if I leave out -i 1-10 (or whatever) it goes through the full list of indexes fine.


ps - sorry if there are any misspelled commands, my workstation running the analysis isn't currently connected to the web, so I'm copying by eye!
seeker is offline   Reply With Quote
Old 08-30-2011, 04:53 AM   #12
seeker
Member
 
Location: Zurich

Join Date: Jan 2011
Posts: 26
Default

actually, it failed again with my real data..(above was with a small test file). any suggestions here would be great.
seeker is offline   Reply With Quote
Old 08-30-2011, 01:04 PM   #13
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by seeker View Post
actually, it failed again with my real data..(above was with a small test file). any suggestions here would be great.
I would need a way to reproduce:
https://sourceforge.net/apps/mediawi...a_help_request

I would be happy to help in that case.
nilshomer is offline   Reply With Quote
Old 08-31-2011, 02:04 AM   #14
seeker
Member
 
Location: Zurich

Join Date: Jan 2011
Posts: 26
Default

It looks like I was doing something stupid by telling it to use more threads than made sense. Reducing that to the number of cores on my processor fixed the problem.
seeker is offline   Reply With Quote
Reply

Tags
bfast

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 03:23 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2021, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO