Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    That's really strange. It actually should have worked fine whether you set it to t=4, t=8, or t=20, but at least it was successful now. Maybe an OS reinstall is not a bad idea.

    Comment


    • #17
      Hi Brian,

      I tried to run BBmerge using this command: java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq out=merged_by_mapping.fq int usemapping parsecustom

      And it spit this at me without generating an output:

      BBMerge version 4.3
      Set INTERLEAVED to true
      Setting FASTQ.PARSE_CUSTOM to true
      Writing mergable reads merged.
      Started output threads.
      Note: Filename indicates non-synthetic data, but FASTQ.PARSE_CUSTOM=true
      Exception in thread "Thread-4" java.lang.AssertionError:
      Mismatch between length of bases and qualities for read 75 (id=412_chr1_0_157870
      44_15787280_-32_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p_
      _Glomeromycota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s_
      _Glomeraceae_sp /2).
      # qualities=412, # bases=229

      FGHHFGGFFGGHGFGHFFHEGGGGGHEHHHHHHGHHHHGEGGGGFHGGGGG@412_chr1_1_15787336_15787565
      _260_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p__Glomeromyc
      ota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s__@413_chr1_
      0_10703971_10704200_-32_Ascomycota_sp|FJ708614|SH229315.06FU|reps|k__Fungi;p__As
      comycota;c__unidentified;o__unidentified;f__unidentified;g__unidentified;s__Asco
      mycota_sp /2
      TTTCCGTAGGTGAACCTGCGGAAGGATCATTACTGATTTGGCAAACCGAGCGTTAGCAAGGTTTTGCGATCAACTTATAT
      TTAAAACCCACTCTTTATATGTATATTTTTTATTTTGAATGATAATTAAAGATCACTTTCAACAACGGATCTCTTGGCTC
      TCGCATCGATGAAGAACGTAGCGACGTGCGATAAGTAATGTGAATTGCAGAATTCCGTGAATCATCGAA

      at stream.Read.<init>(Read.java:72)
      at stream.Read.<init>(Read.java:45)
      at stream.FASTQ.quadToRead(FASTQ.java:756)
      at stream.FASTQ.toReadList(FASTQ.java:620)
      at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:124)

      at stream.FastqReadInputStream.hasMore(FastqReadInputStream.java:80)
      at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(Concurre
      ntGenericReadInputStream.java:739)
      at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGene
      ricReadInputStream.java:731)

      Comment


      • #18
        Hmmm, it's working fine for me.

        I think the problem is that in your command line you are only specifying one pair for the merging. That's fine if the output from BBMap was interleaved, but not if output was in two files. Try this, instead:

        java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in1=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq in2=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R2.fq out=merged_by_mapping.fq usemapping parsecustom

        If that still does not work, please post the first 8 lines of each of the mapped files. It's possible that there is something odd in the headers that is messing with the parsing.

        Comment


        • #19
          Ugh, sorry I made a sloppy copy/paste error. It worked!

          I looked at some quality stats for the final merged file in UPARSE and I noticed that instead of gaps separating the 5' and 3' fragments, there are N's.

          i.e. ATCGTCGAATGCNNNNNNNNNNNNNNCACATTACGAGTTAC

          This is making quality filtering impossible as UPARSE thinks every sequence is loaded with bad base calls. I thought of just doing a find/replace command to change N's to gaps but I'm afraid it may mess with the headers... What are your thoughts?

          Something else that I didn't consider is that since ITS1 is so length variable between taxa, the variable gap size will make end trimming everything to the same length difficult. UPARSE recommends doing this to prevent spurious OTUs.

          Anyway... thanks again for your help! Looks like it worked as advertized

          Comment


          • #20
            Jeff,

            I suggest doing quality filtering prior to merging; otherwise the problem with the Ns will be inevitable. For example:
            java -ea -Xmx200m -cp /path/ jgi.ReformatReads in1=mapped1.fq in2=mapped2.fq out1=filtered1.fq out2=filtered2.fq maq=15

            ...where "maq" means "min average quality" (phred scaled) and reads with an average quality below that will be discarded. Pairs will be either discarded or retained together. Then just merge the filtered reads.

            Comment


            • #21
              OK sounds good, I was worried that filtering prior to merging would change the read orders in the two files... complicating the merging process

              Comment


              • #22
                Originally posted by Brian Bushnell View Post
                I've uploaded a new version of BBTools; you can merge them like this:

                bbmap.sh -Xmx7g ref=reference.fa in=reads.fq outm=mapped.fq outu=unmapped.fq nodisk po int rbm don

                bbmerge.sh in=mapped.fq out=merged_by_mapping.fq int usemapping parsecustom

                bbmerge.sh in=unmapped.fq out=merged_by_overlap.fq outu=unmerged.fq int

                Then you can concatenate the merged_by_mapping.fq and merged_by_overlap.fq files to get a single file.

                Note that these commands all assume you are using a single interleaved file. For two files, you can use the "in1", "in2", "out1", and "out2" flags.
                Is this still the same workflow or is there a simpler process? I have amplicons that are about 634 bp in length, and have read1/read2 sequences that are about 300bp in length, hence there is about a ~34bp gap between read1/read2. I would like to merge them with the gaps but so far finding a piece of software that can do this has been very difficult.

                Comment


                • #23
                  Originally posted by Illusive Man View Post
                  Is this still the same workflow or is there a simpler process? I have amplicons that are about 634 bp in length, and have read1/read2 sequences that are about 300bp in length, hence there is about a ~34bp gap between read1/read2. I would like to merge them with the gaps but so far finding a piece of software that can do this has been very difficult.
                  That workflow should still be valid (although I have not run it). Basically (1) map the reads, (2) merge the reads that map, (3) merge the reads that do not map. I do not think that the method could be made more simple.

                  Comment


                  • #24
                    BBMAP error?

                    Toms-Mac-Pro:bbmap tomhanks$ bash bbmap.sh -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
                    java -Djava.library.path=/Users/bubba/Downloads/bbmap/jni/ -ea -Xmx10g -cp /Users/bubba/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
                    Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -Xmx10g, ref=amoa_archaea.fasta, in1=Read1.fastq, in2=Read2.fastq, outm1=mapped_R1.fq, outu1=unmapped_R1.fq, outm2=mapped_R2.fq, outu2=mapped_R2.fq, nodisk, po, int, rbm, don]

                    BBMap version 35.85
                    Set INTERLEAVED to true
                    Retaining first best site only for ambiguous mappings.
                    Executing dna.FastaToChromArrays2 [amoa_archaea.fasta, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=false, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=true]

                    Set genScaffoldInfo=true
                    Reset INTERLEAVED to false because paired input files were specified.
                    Set genome to 1

                    Loaded Reference: 0.005 seconds.
                    Loading index for chunk 1-1, build 1
                    Indexing threads started for block 0-1
                    Indexing threads finished for block 0-1
                    Generated Index: 0.481 seconds.
                    Analyzed Index: 4.322 seconds.
                    Started output stream: 0.016 seconds.
                    Started output stream: 0.001 seconds.
                    Cleared Memory: 0.147 seconds.
                    Processing reads in paired-ended mode.
                    Started read stream.
                    Started 8 mapping threads.
                    Exception in thread "Thread-8" Exception in thread "Thread-7" java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
                    at stream.FASTQ.customID(FASTQ.java:294)
                    at stream.FASTQ.toFASTQ(FASTQ.java:359)
                    at stream.Read.toFastq(Read.java:549)
                    at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:429)
                    at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
                    at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
                    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
                    java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
                    at stream.FASTQ.customID(FASTQ.java:294)
                    at stream.FASTQ.toFASTQ(FASTQ.java:359)
                    at stream.Read.toFastq(Read.java:549)
                    at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:404)
                    at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
                    at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
                    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
                    Exception in thread "Thread-11" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    Detecting finished threads: 0Exception in thread "Thread-16" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    Exception in thread "Thread-14" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    Exception in thread "Thread-17" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    Exception in thread "Thread-13" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    Exception in thread "Thread-12" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    , 1, 2, 3Exception in thread "Thread-15" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    , 4, 5, 6Exception in thread "Thread-18" java.lang.RuntimeException: Writing to a terminated thread.
                    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
                    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
                    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
                    at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
                    at align2.AbstractMapThread.run(AbstractMapThread.java:551)
                    , 7

                    **************************************************************************
                    Warning! 8 mapping threads did not terminate normally.
                    Check the error log; the output may be corrupt or incomplete.
                    Please submit the full stderr output as a bug report, not just this message.
                    **************************************************************************




                    ------------------ Results ------------------

                    Genome: 1
                    Key Length: 13
                    Max Indel: 16000
                    Minimum Score Ratio: 0.56
                    Mapping Mode: normal
                    Reads Used: 3600 (1064189 bases)

                    Mapping: 14.865 seconds.
                    Reads/sec: 242.19
                    kBases/sec: 71.59


                    Pairing data: pct reads num reads pct bases num bases

                    mated pairs: 5.0556% 91 5.0974% 54246
                    bad pairs: 0.0000% 0 0.0000% 0
                    insert size avg: 553.14


                    Read 1 data: pct reads num reads pct bases num bases

                    mapped: 5.0556% 91 5.0919% 27123
                    unambiguous: 5.0556% 91 5.0919% 27123
                    ambiguous: 0.0000% 0 0.0000% 0
                    low-Q discards: 0.0000% 0 0.0000% 0

                    perfect best site: 0.0000% 0 0.0000% 0
                    semiperfect site: 0.0000% 0 0.0000% 0
                    rescued: 0.0000% 0

                    Match Rate: NA NA 36.8918% 23738
                    Error Rate: 18.6858% 91 61.8649% 39807
                    Sub Rate: 18.6858% 91 4.0174% 2585
                    Del Rate: 8.4189% 41 57.8475% 37222
                    Ins Rate: 0.0000% 0 0.0000% 0
                    N Rate: 18.4805% 90 1.2433% 800


                    Read 2 data: pct reads num reads pct bases num bases

                    mapped: 5.0556% 91 5.0973% 27093
                    unambiguous: 5.0556% 91 5.0973% 27093
                    ambiguous: 0.0000% 0 0.0000% 0
                    low-Q discards: 0.1667% 3 0.0198% 105

                    perfect best site: 0.0000% 0 0.0000% 0
                    semiperfect site: 0.0000% 0 0.0000% 0
                    rescued: 0.0556% 1

                    Match Rate: NA NA 84.0296% 22788
                    Error Rate: 53.2164% 91 15.9335% 4321
                    Sub Rate: 53.2164% 91 15.6200% 4236
                    Del Rate: 7.6023% 13 0.0959% 26
                    Ins Rate: 9.9415% 17 0.2176% 59
                    N Rate: 0.5848% 1 0.0369% 10
                    Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
                    at align2.BBMap.testSpeed(BBMap.java:486)
                    at align2.BBMap.main(BBMap.java:33)
                    Last edited by Illusive Man; 03-13-2016, 10:53 AM.

                    Comment


                    • #25
                      Can you add an explicit threads= option like below and try again:

                      Code:
                       $ bash bbmap.sh -Xmx10g threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t
                      Last edited by GenoMax; 03-09-2016, 12:33 PM. Reason: correction to threads option

                      Comment


                      • #26
                        Originally posted by GenoMax View Post
                        Can you add an explicit threads= option like below and try again:

                        Code:
                         $ bash bbmap.sh -Xmx10g -threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t
                        BBMap version 35.85
                        Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
                        at align2.AbstractMapper.parse(AbstractMapper.java:627)
                        at align2.AbstractMapper.<init>(AbstractMapper.java:51)
                        at align2.BBMap.<init>(BBMap.java:41)
                        at align2.BBMap.main(BBMap.java:29)

                        Comment


                        • #27
                          Originally posted by Illusive Man View Post
                          BBMap version 35.85
                          Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
                          at align2.AbstractMapper.parse(AbstractMapper.java:627)
                          at align2.AbstractMapper.<init>(AbstractMapper.java:51)
                          at align2.BBMap.<init>(BBMap.java:41)
                          at align2.BBMap.main(BBMap.java:29)
                          My fault. Use threads=4 (no hyphen in front). I will edit the example above.

                          Comment


                          • #28
                            Originally posted by GenoMax View Post
                            My fault. Use threads=4 (no hyphen in front). I will edit the example above.
                            Thanks for your help! I tried that but it gave the same error unfortunately...
                            Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
                            at align2.BBMap.testSpeed(BBMap.java:486)
                            at align2.BBMap.main(BBMap.java:33)

                            Comment


                            • #29
                              @Brian should be along later tonight and he will provide a definitive answer.

                              Comment


                              • #30
                                Originally posted by GenoMax View Post
                                @Brian should be along later tonight and he will provide a definitive answer.
                                Great! I look forward to hearing from him. I don't know if it helps any but I also have Geneious 9 and installed the BBMap plugin...maybe I could run part of the workflow in Geneious 9, then do the rest by running scripts. Anyhow I'll wait to hear back from Brian. Thanks!!

                                Comment

                                Latest Articles

                                Collapse

                                • seqadmin
                                  Strategies for Sequencing Challenging Samples
                                  by seqadmin


                                  Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                  03-22-2024, 06:39 AM
                                • seqadmin
                                  Techniques and Challenges in Conservation Genomics
                                  by seqadmin



                                  The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                  Avian Conservation
                                  Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                  03-08-2024, 10:41 AM

                                ad_right_rmr

                                Collapse

                                News

                                Collapse

                                Topics Statistics Last Post
                                Started by seqadmin, Yesterday, 06:37 PM
                                0 responses
                                7 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, Yesterday, 06:07 PM
                                0 responses
                                7 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 03-22-2024, 10:03 AM
                                0 responses
                                49 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 03-21-2024, 07:32 AM
                                0 responses
                                66 views
                                0 likes
                                Last Post seqadmin  
                                Working...
                                X