Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • BWA multithreading

    Hi all,

    I have BWA installed on a Mac OS 10.6 installation with 8 cores (64 bit machine). I have tried using '-t 8' or '-t 4' to use multiple cores, but it doesn't appear to be working? I have built the program from source, but don't seem to be able to find anything I can change in the Makefile.

    Any help would be much appreciated.

    Thanks

  • #2
    Any thoughts?

    Comment


    • #3
      You do not give sufficient detail (input data, command line) to debug the problem. Thanks!

      Comment


      • #4
        Hey Nils,

        Sorry, didn't realize that. Everything has been run pretty standard - here's an example:

        bwa aln -q 5 -l 32 -k 2 -o 1 -t 4 database.fasta reads.fastq > aligned.sai

        As I said - the '-t 4' parameter is the problem as bwa doesn't appear to be able to use multiple cores on my install - and in fact, when I use -t 4, it will slow down the program compared to if that option isn't used.

        I have tried rebuilding bwa from source (two different versions - - either using 'make' or the 'sh autogen.sh' method - neither works).

        At the end of the day, I guess bwa isn't very suited for my purposes (my data is quite divergent), but the Broad is using bwa as their pipeline, so I will have to try and work with it for now - multi threading would help ;-).

        Comment


        • #5
          Provide more detail and maybe we can help. We run BWA on a 64 core machine and it runs great. It scales nearly perfectly (64 cores = 64X speed up).

          Comment


          • #6
            make fails for bwa?
            What's the error message?

            Make sure you're compiling with -DHAVE_PTHREAD When you type make you should see this "string" as parameter to gcc. If you don't have the pthread library, then it won't link.

            Comment


            • #7
              make appears to be working and I don't see any errors popping up. On a fresh install, this is what is written to the terminal window:

              $make
              make[1]: Nothing to be done for `lib'.
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt_gen.c -o bwt_gen.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD QSufSort.c -o QSufSort.o
              ar -cru libbwtgen.a bwt_gen.o QSufSort.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD utils.c -o utils.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt.c -o bwt.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtio.c -o bwtio.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtaln.c -o bwtaln.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtgap.c -o bwtgap.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD is.c -o is.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bntseq.c -o bntseq.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtmisc.c -o bwtmisc.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtindex.c -o bwtindex.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD stdaln.c -o stdaln.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD simple_dp.c -o simple_dp.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwaseqio.c -o bwaseqio.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwase.c -o bwase.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwape.c -o bwape.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD kstring.c -o kstring.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD cs2nt.c -o cs2nt.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_core.c -o bwtsw2_core.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_main.c -o bwtsw2_main.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_aux.c -o bwtsw2_aux.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt_lite.c -o bwt_lite.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_chain.c -o bwtsw2_chain.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bamlite.c -o bamlite.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD main.c -o main.o
              gcc -g -Wall -O2 -DHAVE_PTHREAD utils.o bwt.o bwtio.o bwtaln.o bwtgap.o is.o bntseq.o bwtmisc.o bwtindex.o stdaln.o simple_dp.o bwaseqio.o bwase.o bwape.o kstring.o cs2nt.o bwtsw2_core.o bwtsw2_main.o bwtsw2_aux.o bwt_lite.o bwtsw2_chain.o bamlite.o main.o -o bwa -lm -lz -lpthread -Lbwt_gen -lbwtgen

              My computer is a Mac Pro running 10.6.8 and as far as I can tell pthread should be there. I just use the standard Makefile with no modifications - everything appears to be in order - here's a copy of the file:

              CC= gcc
              CXX= g++
              CFLAGS= -g -Wall -O2
              CXXFLAGS= $(CFLAGS)
              DFLAGS= -DHAVE_PTHREAD #-D_FILE_OFFSET_BITS=64
              OBJS= utils.o bwt.o bwtio.o bwtaln.o bwtgap.o is.o \
              bntseq.o bwtmisc.o bwtindex.o stdaln.o simple_dp.o \
              bwaseqio.o bwase.o bwape.o kstring.o cs2nt.o \
              bwtsw2_core.o bwtsw2_main.o bwtsw2_aux.o bwt_lite.o \
              bwtsw2_chain.o bamlite.o
              PROG= bwa
              INCLUDES=
              LIBS= -lm -lz -lpthread -Lbwt_gen -lbwtgen
              SUBDIRS= . bwt_gen

              .SUFFIXES:.c .o .cc

              .c.o:
              $(CC) -c $(CFLAGS) $(DFLAGS) $(INCLUDES) $< -o $@
              .cc.o:
              $(CXX) -c $(CXXFLAGS) $(DFLAGS) $(INCLUDES) $< -o $@

              all:$(PROG)

              lib-recur all-recur clean-recur cleanlocal-recur install-recur:
              @target=`echo $@ | sed s/-recur//`; \
              wdir=`pwd`; \
              list='$(SUBDIRS)'; for subdir in $$list; do \
              cd $$subdir; \
              $(MAKE) CC="$(CC)" CXX="$(CXX)" DFLAGS="$(DFLAGS)" CFLAGS="$(CFLAGS)" \
              INCLUDES="$(INCLUDES)" $$target || exit 1; \
              cd $$wdir; \
              done;

              lib:

              bwa:lib-recur $(OBJS) main.o
              $(CC) $(CFLAGS) $(DFLAGS) $(OBJS) main.o -o $@ $(LIBS)

              bwt.o:bwt.h
              bwtio.o:bwt.h
              bwtaln.o:bwt.h bwtaln.h kseq.h
              bwt1away.o:bwt.h bwtaln.h
              bwt2fmv.o:bwt.h
              bntseq.o:bntseq.h
              bwtgap.o:bwtgap.h bwtaln.h bwt.h

              bwtsw2_core.o:bwtsw2.h bwt.h bwt_lite.h stdaln.h
              bwtsw2_aux.o:bwtsw2.h bwt.h bwt_lite.h stdaln.h
              bwtsw2_main.o:bwtsw2.h

              cleanlocal:
              rm -f gmon.out *.o a.out $(PROG) *~ *.a

              clean:cleanlocal-recur

              Comment


              • #8
                Just in case anybody else ran into this problem. The lack of multi-threading was caused by the size of the reference file I was using - my genome is only 4kb, so I guess bwa couldn't throw all the data at 8 cores - this just slowed down the program, rather than speeding it up. Using hg19 as a reference, everything is working great.

                Comment

                Latest Articles

                Collapse

                • seqadmin
                  Strategies for Sequencing Challenging Samples
                  by seqadmin


                  Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                  03-22-2024, 06:39 AM
                • seqadmin
                  Techniques and Challenges in Conservation Genomics
                  by seqadmin



                  The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                  Avian Conservation
                  Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                  03-08-2024, 10:41 AM

                ad_right_rmr

                Collapse

                News

                Collapse

                Topics Statistics Last Post
                Started by seqadmin, Yesterday, 06:37 PM
                0 responses
                8 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, Yesterday, 06:07 PM
                0 responses
                8 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 03-22-2024, 10:03 AM
                0 responses
                49 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 03-21-2024, 07:32 AM
                0 responses
                67 views
                0 likes
                Last Post seqadmin  
                Working...
                X