BBMap (aligner for DNA/RNAseq) is now open-source and available for download.

[susanklein]

Originally Posted by GenoMax

Since you are using ambig=all you are going to get multi-mappers with their MAPQ being set to a small value (3). You could post filter your sam file or set ambig=toss (or ambig=random/best to get only one alignment).

ambig=best worked, Thanks.

S.

[susanklein]

Ok I thought this was solved, but see below, still getting sam file alignments with MAPQ=3, even though they actually look like good alignments. Is '3' given because you can't calculate a perfect hit MAPQ? i.e. MAPQ=-10*log10(zero)

@PG ID:BBMap PN:BBMap VN:37.93 CL:java -Djava.library.path=/bin/bbmap/jni/ -ea -Xmx43986m align2.BBMap build=1 overwrite=true fastareadlen=500 ref=genome/genome.fasta ambig=best vslow perfectmode maxsites=10000 nzo=t mappedonly=t outu=test/unmapped/testKT45.unmapped.fasta scafstats=test/mapped/testKT45.map in=testKT45.fastq out=test/mapped/testKT45.sam -idfilter=0.9
700666F:325:CCC50ANXX:6:2211:7729:2233 1:N:0:GTCGAG 0 ctg3849_RHIMB__RM170330.3849 Backbone_3849|arrow|pilon 111580 3 31= * 0 0 GCATTGGTGGTTCAGTGGTAGAATTCTCGCC CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGG XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7914:2047 1:N:0:GTAGAG 16 ctg18961_RHIMB__RM170330.18961 Backbone_18961|arrow|pilon 20097 3 31= * 0 0 TCACAATGATAGGAAGAGCCGACATCGAAGG GGGGGGGGGGGGGGGGGGGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7905:2066 1:N:0:GTCGAG 16 ctg18961_RHIMB__RM170330.18961 Backbone_18961|arrow|pilon 34735 3 31= * 0 0 GGTAGGCGCAGAAAGTACCATCGAAAGTTGA GGGGGFGGGGGGGGGGGGGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7970:2099 1:N:0:GTCGAG 0 ctg16585_RHIMB__RM170330.16585 Backbone_16585|arrow|pilon 33974 3 23= * 0 0 GGGAATACCAGGTGCTGTAGGCT CCCCCGEGGGGGGGGGGGGGGGG XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7937:2145 1:N:0:GTCGAG 16 ctg13544_RHIMB__RM170330.13544 Backbone_13544|arrow|pilon 7627 3 39= * 0 0 GGTGAGAGCGCCGAATCCTAACCACTAGACCACCAGGGA GGGGGGGGGGGGGGGGGGGGGGGGGFGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3

[darthsequencer]

Hi Brian - Is there a bbtool that will keep only mapped reads and if the mate doesn't map output the unmapped also from sam/bam files?

Thanks

[Brian Bushnell]

Originally Posted by susanklein

Ok I thought this was solved, but see below, still getting sam file alignments with MAPQ=3, even though they actually look like good alignments. Is '3' given because you can't calculate a perfect hit MAPQ? i.e. MAPQ=-10*log10(zero)

@PG ID:BBMap PN:BBMap VN:37.93 CL:java -Djava.library.path=/bin/bbmap/jni/ -ea -Xmx43986m align2.BBMap build=1 overwrite=true fastareadlen=500 ref=genome/genome.fasta ambig=best vslow perfectmode maxsites=10000 nzo=t mappedonly=t outu=test/unmapped/testKT45.unmapped.fasta scafstats=test/mapped/testKT45.map in=testKT45.fastq out=test/mapped/testKT45.sam -idfilter=0.9
700666F:325:CCC50ANXX:6:2211:7729:2233 1:N:0:GTCGAG 0 ctg3849_RHIMB__RM170330.3849 Backbone_3849|arrow|pilon 111580 3 31= * 0 0 GCATTGGTGGTTCAGTGGTAGAATTCTCGCC CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGG XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7914:2047 1:N:0:GTAGAG 16 ctg18961_RHIMB__RM170330.18961 Backbone_18961|arrow|pilon 20097 3 31= * 0 0 TCACAATGATAGGAAGAGCCGACATCGAAGG GGGGGGGGGGGGGGGGGGGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7905:2066 1:N:0:GTCGAG 16 ctg18961_RHIMB__RM170330.18961 Backbone_18961|arrow|pilon 34735 3 31= * 0 0 GGTAGGCGCAGAAAGTACCATCGAAAGTTGA GGGGGFGGGGGGGGGGGGGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7970:2099 1:N:0:GTCGAG 0 ctg16585_RHIMB__RM170330.16585 Backbone_16585|arrow|pilon 33974 3 23= * 0 0 GGGAATACCAGGTGCTGTAGGCT CCCCCGEGGGGGGGGGGGGGGGG XT:A:R NM:i:0 AM:i:3
700666F:325:CCC50ANXX:6:2211:7937:2145 1:N:0:GTCGAG 16 ctg13544_RHIMB__RM170330.13544 Backbone_13544|arrow|pilon 7627 3 39= * 0 0 GGTGAGAGCGCCGAATCCTAACCACTAGACCACCAGGGA GGGGGGGGGGGGGGGGGGGGGGGGGFGGGGGGGGCCCCC XT:A:R NM:i:0 AM:i:3

MAPQ is a measure of the probability (estimate) that the mapping location is correct. This can factor in various things, including the number of mismatches... but, for example, with these very short sequences, even though they are perfect matches, the specificity is low. MAPQ=3 specifically means that there is at most a 50% chance that the mapping is correct, and thus, there's at least one other perfect alignment for each of these.

[Brian Bushnell]

Originally Posted by darthsequencer

Hi Brian - Is there a bbtool that will keep only mapped reads and if the mate doesn't map output the unmapped also from sam/bam files?

Thanks

Not exactly. reformat.sh has a "mappedonly" flag, but that would only keep the mapped reads. However, you can use requiredbits and filterbits in 2 passes, like this:

reformat.sh in=data.sam out=mapped.sam mappedonly
reformat.sh in=data.sam out=unmapped_mates_of_mapped_reads.sam requiredbits=4 filterbits=8

Then merge the two output files. Alternatively:

reformat.sh in=data.sam out=mapped.sam mappedonly
filterbyname.sh in=data.sam names=mapped.sam include=t out=mapped_or_mate_mapped.sam

The second solution takes more memory but it keeps the reads in the input order and is generally more convenient.

[rulix]

Hi Brian! Hope I'm posting this in the right place.

I've been using clumpify for a while. Great tool! I was totally blown away by how fast it is...

Simple question. I noticed that you make the following recommendations for detection of optical duplicates:

dupedist=40 (dist) Max distance to consider for optical duplicates.
Higher removes more duplicates but is more likely to
remove PCR rather than optical duplicates.
This is platform-specific; recommendations:
NextSeq 40 (and spany=t)
HiSeq 1T 40
HiSeq 2500 40
HiSeq 3k/4k 2500
Novaseq 12000

I was surprised to see that NovaSeq has such a big distance. Is this something you have measured? I have only been able to find the Picard recommendation which groups all the patterned flowcells with the same distance (2500).

Thanks!

[Brian Bushnell]

The coordinate system varies across platforms; the physical distance of 1 pixel is much larger on HiSeq 2500 than HiSeq 3000/4000m for example.

NovaSeq has unique problems, though. The distance of 12000 is not really to catch optical duplicates, but rather, clonal colonies that presumably form when a molecule breaks off from a colony and forms a new colony nearby (typically downstream). These have a high degree of positional correlation but are not actually adjacent; they can be fairly distant. They are not even necessarily on the same tile, so Clumpify won't catch all of them when it is restricted to looking for duplicates only within a tile.

And yes, the 12000 comes from empirical testing. That catches most of the duplicates formed by this mechanism. I think I posted the approximate amount somewhere... maybe it was around 85%? You can't get all of them unless you completely disregard position and remove all duplicates.

[GenoMax]

We had a discussion about what the x:y coordinates mean on Biostars.

Short take home from poster of that thread was:

Contacted illumina support and they said the information I needed, such as dimension of the tile viewing area or zoom of the microscope, was "considered proprietary," which is frustrating.

[susanklein]

Originally Posted by Brian Bushnell

MAPQ is a measure of the probability (estimate) that the mapping location is correct. This can factor in various things, including the number of mismatches... but, for example, with these very short sequences, even though they are perfect matches, the specificity is low. MAPQ=3 specifically means that there is at most a 50% chance that the mapping is correct, and thus, there's at least one other perfect alignment for each of these.

ah I see, thanks. So they are good alignments, but to multiple sites. I'll have to rethink my strategy because I really should be keeping all alignments.

Thanks,

S.

[GenoMax]

Originally Posted by susanklein

ah I see, thanks. So they are good alignments, but to multiple sites. I'll have to rethink my strategy because I really should be keeping all alignments.

Thanks,

S.

Consider http://seqanswers.com/forums/showthread.php?p=214346 (posts 204-206). BBMap does not seem to report ALL alignments in output file. @Brian may seen this and comment.

[kbseah]

(Apologies if this shows up more than once, my posts keep disappearing...)
Hello Brian et al.,

The bitflags don't seem to be assigned correctly when the ambig=all option is used. The renaming of read pairs with keepnames=f also appears to be affected.

I am mapping Illumina paired reads with ambig=all to report all ambiguous alignments. From what I understand, the secondary alignments are indicated with the bitflag 0x100. For paired input, the alignments appear to be reported in the following order for each read pair:

• read F primary alignment
• read F secondary alignment(s)
• read R primary alignment
• read R secondary alignment(s)

I observed that with keepnames=f, there are two bugs (example below):

• Reverse read is renamed to match the forward read only for the primary alignment. The original name (e.g. with the tag 2:N in the example below) is retained in the secondary alignment(s)
• Secondary alignments of the reverse read are erroneously given bitflag 0x40 (first segment in the template, i.e. forward read) instead of 0x80 (last segment of the template, i.e. reverse read)

With keepnames=t, the original names are retained for both forward and reverse reads in both primary and secondary alignments, but the second bug above (misassignment of bitflags 0x40 and 0x80) is still there.

I first noticed the problem with bbmap v37.76 but it still occurs in v37.99.

Maybe I'm overlooking something, but this seems to be a bug. Has anyone else encountered this before? My apologies if this has been posted to this thread before. I didn't find anything relevant with search keywords 'keepnames' and 'ambiguous'.

Thank you!

** Examples **

Alignment columns QNAME and FLAG from SAM file with keepnames=f for a read pair showing the error

HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	99
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	147
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337

Alignment columns QNAME and FLAG from SAM file with keepnames=t for the same read pair as above

HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	99
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 1:N:0:AAGGGAAT	353
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	147
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337
HWI-ST863:279:H03F7ADXX:1:1107:5049:39482 2:N:0:AAGGGAAT	337

[denisDDS]

Dear BBMap programmers/users,

First of all, I want to thank you for your work. This set of tools is very pleasant to use and very well documented. Moreover, the pipeline are very useful and very well explained. This is very easy easy to adapt them.

Nevertheless, as this is the first time I am using BBMap, on a really specific problem, I have few questions (you can find an explanation of my problem here: http://seqanswers.com/forums/showthread.php?t=81997).
I used the variantPipeline.sh pipeline adapted for multiple paired non-covering amplicon data files:
- Does BBMap support IUPAC nucleotide code for building the reference genome index?
- As I will map distant (or polymorph) reads on my reference, I used "k=8" for building the index. is it possible to lower this value and is it useful?
- filterbytile.sh, for trimming the adaptater does not take input file describing my adapters (Nextera...)? Is it normal?
- bbduk.sh, removing the sequencing artifact, take default files (ref=${BBMAPRESSOURCES}/sequencing_artifacts.fa.gz,${BBMAPRESSOURCES}/phix174_ill.ref.fa.gz). Is it commonly found artefacts? Could I use these files or must I create mine? I didn't change k=27, must I change the value due to the poor quality of my reads?
- bbmap.sh, for mapping step, I used "vslow k=8 maxindel=200 minratio=0.1" parameters, as described in the online documentation (https://jgi.doe.gov/data-and-tools/b...e/bbmap-guide/). Is it a good idea with distant reference genome and poor quality data?
- callvariants.sh, for the variants call step, I let the ploidy=1. I'm working on various plants and I don't really know the ploidy. Must I let this parameter to 1? I'm only looking for SNPs, Is it possible to specify it to the scripts?
- Finally, is it really useful to add optional error-correction and recalibration step of my reads, before performing the mapping? Will I lose some important data?

I know some questions are trivial, but I'm still reading the manual and the different scripts documentations...

Thanks in advance for your help.

Denis

[bwubb]

Originally Posted by GenoMax

BBMap is one of the aligners that uses the full header present in your fasta file when creating the index and passes it along to alignment file. If there are spaces in the header name they are written to alignment. Some downstream programs have a problem with this.

You can use the option "trd=t" to truncate the fasta header names after the first space in the name. There is an option for reformat.sh that can do this after the fact for aligned data. I can look this up later if you don't find it.

Thank you. I had found this too sometime after my post and before yours, but its really great to get a helpful reply!

[Gopo]

Hi Brian,

Does bbmap callvariants.sh ignore duplicates marked by picard MarkDuplicates by default (or is there an option to ignore duplicates) or do duplicates have to be deleted?

Best,
Gopo

[darthsequencer]

Hi Brian,
I'm using trimrname=t from reformat.sh on a sam file. It looks like it trims the ref names correctly on the alignment lines, but not in header lines (the ones that start with @).

[Meyana]

Dear Brian,

I am using bbmap for mapping and callvariants.sh for variant calling on PE Illumina reads.
I am comparing two mice strains. I am therefore downsampling my .sam files to have the same number of mapped reads going into the alignment.

When I input the original file containing ~680k mapped PE reads I get 2283 variants. When I run the down sampled file containing ~200k mapped PE reads I get 3375 variants.

I would expect to get a lower number of variants when I put in less reads. Could you explain this to me? I am clearly missing something that might be important for my analysis

(I am using the exact same criteria for variant calling and filtering for the two samples)

[gaohanlisa]

I am trying to run BBmap on the cluster, but got the error below. Can anyone help me to solve the error?
Thanks,
[hgx080@quser10 DNA_all]$ /home/hgx080/bbmap/bbmap.sh ref=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/assemble/final/idba/DNA-all/DNA-all-contig.fa in=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/clean_reads/Wells02/filter_reads/RNA-Ac-1_S7_filter.fa out=RNA-Ac-1_S7_filter.test.sam minid=0.95 ambig=random reads=100000 -Xmx100g -eoom
java -Djava.library.path=/home/hgx080/bbmap/jni/ -ea -Xmx100g -cp /home/hgx080/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/assemble/final/idba/DNA-all/DNA-all-contig.fa in=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/clean_reads/Wells02/filter_reads/RNA-Ac-1_S7_filter.fa out=RNA-Ac-1_S7_filter.test.sam minid=0.95 ambig=random reads=100000 -Xmx100g -eoom
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/assemble/final/idba/DNA-all/DNA-all-contig.fa, in=/projects/b1052/Wells_b1042/GaoHan/CANDO_RNA/clean_reads/Wells02/filter_reads/RNA-Ac-1_S7_filter.fa, out=RNA-Ac-1_S7_filter.test.sam, minid=0.95, ambig=random, reads=100000, -Xmx100g, -eoom]
Version 38.11

Choosing a site randomly for ambiguous mappings.
Set MINIMUM_ALIGNMENT_SCORE_RATIO to 0.908
NOTE: Ignoring reference file because it already appears to have been processed.
NOTE: If you wish to regenerate the index, please manually delete ref/genome/1/summary.txt
Max reads: 100000
Set genome to 1

Exception in thread "Thread-0" java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException: Unexpected end of ZLIB input stream
at align2.ChromLoadThread.run(ChromLoadThread.java:79)
Caused by: java.lang.RuntimeException: java.io.EOFException: Unexpected end of ZLIB input stream
at fileIO.ReadWrite.readObject(ReadWrite.java:806)
at fileIO.ReadWrite.read(ReadWrite.java:1246)
at dna.ChromosomeArray.read(ChromosomeArray.java:65)
at align2.ChromLoadThread.run(ChromLoadThread.java:76)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2620)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:3031)
at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:3061)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1914)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at fileIO.ReadWrite.readObject(ReadWrite.java:802)
... 3 more

[StephCarr]

Hello! Thanks for all of the wonderful bbmap scripts. Today I was was trying to use bbfakereads.sh but the script can not locate or open the jgi/FakeReads file. Any thoughts? I noticed a Fakereads files in the current/jgi directory. Do you think the path in the script is incorrect?

Thanks for your time and help!

bbfakereads.sh in=scaffolds.fasta out=fakePE_R1.fasta out2=fakePE.R2.fasta length=150
java -ea -Xmx600m -cp /media/bioinformaticprograms/BBMap/sh/current/ jgi.FakeReads in=scaffolds.fasta out=fakePE_R1.fasta out2=fakePE.R2.fasta length=150
Error: Could not find or load main class jgi.FakeReads

[GenoMax]

Is your bbmap installed correctly? Have you moved any files around? I am able to run "bbfakereads.sh" and generate fastq and fasta files without a problem.

[StephCarr]

You're right. It wasn't downloaded correctly. At first I used git to download the bbmap package. But when I just downloaded with wget from https://sourceforge.net/projects/bbm...p_38.12.tar.gz everything was organized correctly.

Thanks for the help.