Introducing BBMap, a new short-read aligner for DNA and RNA

[Brian Bushnell]

BBMap will be publicly released soon, pending confirmation with LBL's legal department.

In the meantime feel free to look at these graphs of its performance:

https://drive.google.com/file/d/0B3l...it?usp=sharing

Note that this is a 50MB powerpoint file. It contains graphs of relative performance of BBMap and other short read aligners (bwa, bowtie2, gsnap, smalt) mapping synthetic data.

EDIT:
This thread is now closed; please use this one to post questions.

[kopi-o]

Looks very impressive! Can it beat STAR (speed and accuracy wise) for RNA-seq though? (RNA-seq is listed as one of the use cases towards the end)

[Brian Bushnell]

I have compared it to tophat, which it greatly outperforms in speed and has higher sensitivity on real RNA-seq data. I have not yet compared it to STAR - I tried to but was unable to get STAR to run without core-dumping so I gave up. I may have compiled it wrong; I'll try again eventually.

However, I don't have a really good tool for generating and evaluating synthetic RNA-seq data, so it's harder to quantify. The closest I can get is to generate synthetic DNA reads with very large deletions, which is not quite the same thing since RNA-seq data has other strange artifacts and the introns are not distributed randomly.

[dpryan]

It'd be great if you could get in touch with the authors of this paper and just use their test datasets. That would allow comparisons against most of the popular aligners out there.

[GenoMax]

Originally Posted by dpryan

It'd be great if you could get in touch with the authors of this paper and just use their test datasets. That would allow comparisons against most of the popular aligners out there.

Data is available here:

http://www.ebi.ac.uk/arrayexpress/ex...s/E-MTAB-1728/
http://www.ebi.ac.uk/arrayexpress/ex...-1728/samples/

[Brian Bushnell]

Originally Posted by dpryan

It'd be great if you could get in touch with the authors of this paper and just use their test datasets. That would allow comparisons against most of the popular aligners out there.

Thanks for the suggestion; I'll look into that!

[dietmar13]

why is RUM always neglected by comparing RNA-seq mappers?
In my hands RUM outperforms other pipelines, e.g. tophat, in sensitivity, especially for spliced reads...

RUM: RNA Seq Unified Mapper
https://github.com/itmat/rum/wiki

RUM is rather slow, but using multithreaded servers allows mapping in tolerable time (compared to sample and library generation and data interpretation)

dietmar

[Corydoras]

Hi Brian,

I got a file with cleaned sequence data and I want to assemble this de-novo using velvet. Due to the nature of the sequencing and the library protocol, my kmer coverage is quite variable and I wanted to use BBnorm to normalize the coverage a bit to aid the assembly. Am I correct that BBnorm is the right thing to use for this?

Anyway, currently trying to give it a go and I got this error message:

bbmap$ sh bbnorm.sh in=Fowleri_combined.fastq out=normFowleri.fastq target=15
bbnorm.sh: 104: bbnorm.sh: Bad substitution
bbnorm.sh: 112: bbnorm.sh: [[: not found
bbnorm.sh: 112: bbnorm.sh: [[: not found
bbnorm.sh: 118: bbnorm.sh: source: not found
bbnorm.sh: 119: bbnorm.sh: parseXmx: not found
bbnorm.sh: 120: bbnorm.sh: [[: not found
bbnorm.sh: 123: bbnorm.sh: freeRam: not found
java -ea -Xmxm -cp /home/martin/Downloads/bbmap/current/ jgi.KmerNormalize bits=32 in=Fowleri_combined.fastq
Invalid maximum heap size: -Xmxm
Could not create the Java virtual machine.

Any ideas?

Many thanks,

Sarah

[Brian Bushnell]

Sarah,

Yes, BBNorm is the correct tool.

I'm not sure, but I suspect that your shell is not bash. You could retry the command with "bash" instead of "sh", which may work. But the easier thing is just to skip the shellscript and invoke java manually:

java -ea -Xmx14g -cp /home/martin/Downloads/bbmap/current/ jgi.KmerNormalize bits=32 in=Fowleri_combined.fastq out=normFowleri.fastq target=15

That command would work if you had 16g of RAM. Just set the -Xmx parameter (highlighted in purple) to about 85% of however much RAM is on the machine. If you don't know, you should be able to find out like this on a Linux system:

cat /proc/meminfo

...then look at the first line, "MemTotal".

However, 15x is a fairly low target depth. For velvet I would suggest at least 30x for an optimal assembly, unless you just don't have enough data.

-Brian

[Corydoras]

Hi Brian,

That worked like a charm, thank you! The normalization also greatly improved the assemblies and the kmer-coverage distribution looks much nicer. I was just wondering: by default, bbnorm will use a kmer of 31. But for my assembly I am using 41. The assembly works fine, but is it advisable to normalize the coverage using a kmer of 41?

Thanks,

Sarah

[Brian Bushnell]

Sarah,

It might be better to normalize using a kmer length of 41, but BBNorm only supports a maximum of 31 In practice, it should make very little difference, though. Using long kmers is important for assembly, as it helps span short repeats that would otherwise cause contigs to terminate. But normalization is much less sensitive to that issue, and very long kmers can cause problems in the presence of errors. With k=31, a 100bp read with 1 error could yield 31 kmers with a depth of 1, out of a total of 70 kmers - in that case, the median depth would not be impacted. With k=63, there could be 63 of the 70 total kmers spanning the error, thus having a depth of 1, so the median depth of the read would look like 1 instead of its correct value. And BBNorm normalizes based on the median kmer depth of a read.

It's a lot more computationally efficient to use a max kmer length of 31, so that's how I designed it. I've tried shorter kmers down to about k=25 and not noticed an appreciable difference in normalization or error correction.

As for your prior (deleted) post, sorry for not responding - I think the problem was that you were running Java 6 instead of Java 7. Most of the programs in BBTools work fine in Java 6 but it looks like BBNorm requires Java 7 (or higher).

[Corydoras]

Hi Brian,

Thanks so much for that explanation . I thought I wouldn't be able to go past 31 but it is best to double check.

Sorry as well for just deleting my post (and bombarding you with simple questions, new to the world of NGS!), I played around with updating the Java on our Linux machine and that did the trick .

Thanks again for your help! And the fantastic and easy to use script!!

Sarah

[muol]

Hi Brian,

Is there an option to set read quality encoding in bbnorm? I had to set qin=33 in bbduk for some Illumina 1.9 paired end libraries, but this option doesn't seem to exist in bbnorm (used BBMap v. 32.32 for Java 7).

Thanks
Olaf

[Brian Bushnell]

Olaf,

It's there, I just forgot to document it; sorry! I'll add that to the shellscript in the next release. I think that all of the programs in the package that read fastq input allow the "qin" flag.

-Brian

[muol]

Indeed, just tried it and it works well with bbnorm.

Thanks
Olaf

[muol]

Brian,

I ran into a smaller issue with bbnorm. When trying to input and output separate files for a PE library like this:

bbnorm.sh in1=R1.fastq.gz in2=R2.fastq.gz out1=R1.bbnorm.fastq.gz out2=R2.bbnorm.fastq.gz prefilter=t tossbadreads=t ecc=t fixspikes=t qin=33 -Xmx72g target=40

I receive this error during pass 2:

Exception in thread "main" java.lang.AssertionError: Please do not set 'interleaved=true' with dual input files.
	at stream.ConcurrentGenericReadInputStream.<init>(ConcurrentGenericReadInputStream.java:132)
	at stream.ConcurrentGenericReadInputStream.getReadInputStream(ConcurrentGenericReadInputStream.java:661)
	at stream.ConcurrentGenericReadInputStream.getReadInputStream(ConcurrentGenericReadInputStream.java:641)
	at kmer.KmerCount7MTA.countFastq(KmerCount7MTA.java:355)
	at kmer.KmerCount7MTA.makeKca(KmerCount7MTA.java:222)
	at jgi.KmerNormalize.runPass(KmerNormalize.java:1006)
	at jgi.KmerNormalize.main(KmerNormalize.java:736)

Setting interleaved=false doesn't change that. Outputting to a single, interleaved file (in1=xxx in2=xxx out=xxx) on the other hand works fine. Any ideas?

Olaf

[Brian Bushnell]

Olaf,

Currently, BBNorm uses single interleaved files for temporary storage when using multiple passes. And I have not implemented any way to specify dual files in intermediate stages, since everyone at JGI uses interleaved files for everything.

You have two options.
1) You could set "passes=1", which is faster, but I don't recommend it because it doesn't give as good results as 2-pass normalization.
or
2) You could specify only a single output file, which will get interleaved reads:

bbnorm.sh in1=R1.fastq.gz in2=R2.fastq.gz out=R12.bbnorm.fastq.gz prefilter=t tossbadreads=t ecc=t fixspikes=t qin=33 -Xmx72g target=40

...Then, if you need to, de-interleave it afterward:

reformat.sh in=R12.bbnorm.fastq.gz out1=R1.bbnorm.fastq.gz out2=R2.bbnorm.fastq.gz

Sorry for the inconvenience! I'll try to fix that by the next release, though unlike documenting the "qin" flag, this will take more work so no guarantees. Thanks for bringing it to my attention. FYI, the flag "interleaved" has no effect on output, only input.

-Brian

[muol]

Thanks for the info Brian, it wasn't a big issue.

Olaf

[Brian Bushnell]

Olaf,

This has been fixed in the latest release, 33.04

[muol]

Excellent, just did a test run. This is very useful software!

Olaf