SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
understand the "left" and "right" outputs from NCBI data alittleboy Bioinformatics 1 03-14-2014 02:50 AM
DEXSeq error in estimateDispersions: match.arg(start.method, c("log(y)", "mean")) fpadilla Bioinformatics 14 07-03-2013 03:11 PM
The position file formats ".clocs" and "_pos.txt"? Ist there any difference? elgor Illumina/Solexa 0 06-27-2011 08:55 AM
"Systems biology and administration" & "Genome generation: no engineering allowed" seb567 Bioinformatics 0 05-25-2010 01:19 PM
SEQanswers second "publication": "How to map billions of short reads onto genomes" ECO Literature Watch 0 06-30-2009 12:49 AM

Reply
 
Thread Tools
Old 04-30-2014, 03:41 PM   #1
roliwilhelm
Member
 
Location: Vancouver

Join Date: Jun 2012
Posts: 36
Default "Re-inflating Partioned" Metagenomic Data - KHMER

Hello,

My question has two parts. Starting with the most important question:

1) I've read that short-read assemblers designed for metagenomic data make use of read abundance in the assembly process. However, I like the idea of digital normalization (using khmer) as a tool for bringing out reads from genomes which are less abundant in the mix. Is it a wise idea to perform digital normalization and then use assemblers geared for metagenomic data, like MetaVelvet, IDBA_UD or RAY-meta?

2) I have already attempted to partition my metagenomic data using khmer to improve assembly, however found it is very very computationally intensive and slow. The authors of khmer seem to suggest that one viable method would be to perform digital normalization, then partitioning and then "re-inflated" your reads to pre-normalized abundances (see khmer documentation (HERE). I am keen to try that, but the script ("sweep-reads3.py") is no longer comes prepackaged with the new khmer release. I did find it on their git-hub account, HERE, but wonder why it is no longer packaged with the release. Before investing time and energy, I was wondering if anyone has thoughts on this?

Thanks in advance
roliwilhelm is offline   Reply With Quote
Old 04-30-2014, 04:08 PM   #2
Brian Bushnell
Super Moderator
 
Location: Walnut Creek, CA

Join Date: Jan 2014
Posts: 2,695
Default

Quote:
Originally Posted by roliwilhelm View Post
Hello,

My question has two parts. Starting with the most important question:

1) I've read that short-read assemblers designed for metagenomic data make use of read abundance in the assembly process. However, I like the idea of digital normalization (using khmer) as a tool for bringing out reads from genomes which are less abundant in the mix. Is it a wise idea to perform digital normalization and then use assemblers geared for metagenomic data, like MetaVelvet, IDBA_UD or RAY-meta?
I think normalization would be more useful when assembling metagenomic data with a normal assembler. I've found it to improve metagenome assemblies with Soap and Velvet, for example, but have not tried it with metagenome assemblers.

Quote:
2) I have already attempted to partition my metagenomic data using khmer to improve assembly, however found it is very very computationally intensive and slow. The authors of khmer seem to suggest that one viable method would be to perform digital normalization, then partitioning and then "re-inflated" your reads to pre-normalized abundances (see khmer documentation (HERE). I am keen to try that, but the script ("sweep-reads3.py") is no longer comes prepackaged with the new khmer release. I did find it on their git-hub account, HERE, but wonder why it is no longer packaged with the release. Before investing time and energy, I was wondering if anyone has thoughts on this?

Thanks in advance
BBNorm is much faster than khmer, has less bias toward error reads, and it also supports partitioning - rather than normalizing, you can split data into a low coverage bin, medium coverage bin, and high-coverage bin, with custom cutoffs. That would make a lot more sense, computationally.

For example:
bbnorm.sh passes=1 in=reads.fq outlow=low.fq outmid=mid.fq outhigh=high.fq lowbindepth=50 highbindepth=200
That will divide the data into coverage 1-50, 51-199, and 200+.

To normalize to depth 50, the command would be:
bbnorm.sh in=reads.fq out=normalized.fq target=50

Also, there is an alternative to normalization, that works like this:

Downsample to 1% depth.
Assemble.
Map to assembly and keep reads that don't map.
Downsample unmapped reads to 10% depth.
Assemble.
Combine assemblies (with a tool like Dedupe to prevent redundant contigs).
Map to combined assembly.
...
etc.

It's not clear that either is universally better; both approaches have advantages and disadvantages. You can downsample with reformat, also in the BBTools package, like this:
reformat.sh in=reads.fq out=sampled.fq samplerate=0.01

Last edited by Brian Bushnell; 04-30-2014 at 04:28 PM.
Brian Bushnell is offline   Reply With Quote
Old 05-05-2014, 12:59 PM   #3
cjfields
Junior Member
 
Location: Champaign, IL, USA

Join Date: Sep 2009
Posts: 6
Default

Worth mentioning that the partitioning that BBNorm appears to use is coverage-based. khmer uses a (simplified) de Bruijn graph-based partitioning (separating into disconnected partitions of the graph). That's a very important distinction between the two.

Last edited by cjfields; 05-05-2014 at 01:02 PM.
cjfields is offline   Reply With Quote
Old 05-05-2014, 01:24 PM   #4
Brian Bushnell
Super Moderator
 
Location: Walnut Creek, CA

Join Date: Jan 2014
Posts: 2,695
Default

Quote:
Originally Posted by cjfields View Post
Worth mentioning that the partitioning that BBNorm appears to use is coverage-based. khmer uses a (simplified) de Bruijn graph-based partitioning (separating into disconnected partitions of the graph). That's a very important distinction between the two.
Thanks for pointing that out; I was unaware that khmer's partitioning was NOT coverage-based. Yes, BBNorm's partitioning is purely coverage-based and will not be useful except in situations where you have multiple organisms (or organelles, plasmids, etc) with highly different coverage, though that's typically the case in metagenomes.

That said - I've found partitioning by connectivity (overlap, debruijn, etc) in metagenomes to be problematic with short (~100bp) reads; the situation can easily devolve into a single cluster because everything will be connected by a single highly-conserved element, like a 16s subsequence. With longer (~250bp) reads it seems to work better.
Brian Bushnell is offline   Reply With Quote
Old 07-19-2014, 05:35 PM   #5
crusoe
Programmer & Bioinformatician
 
Location: Lansing, MI

Join Date: Oct 2012
Posts: 10
Default

The current and up-to-date method for metagenomic assembly using khmer is kept at http://khmer-protocols.readthedocs.org
crusoe is offline   Reply With Quote
Old 12-04-2014, 05:29 PM   #6
titusbrown
Junior Member
 
Location: Midwest

Join Date: Aug 2013
Posts: 8
Default Reinflation isn't necessary.

Hi all, we now have direct evidence that digital normalization works fine with both SPAdes and IDBA on at least one mock community data set -- neither assembler seems to do poorly on it. I haven't written it up for a blog post yet but I'm happy to send the numbers your way if you're interested.

--titus
titusbrown is offline   Reply With Quote
Old 12-04-2014, 06:41 PM   #7
cjfields
Junior Member
 
Location: Champaign, IL, USA

Join Date: Sep 2009
Posts: 6
Default

Quote:
Originally Posted by titusbrown View Post
Hi all, we now have direct evidence that digital normalization works fine with both SPAdes and IDBA on at least one mock community data set -- neither assembler seems to do poorly on it. I haven't written it up for a blog post yet but I'm happy to send the numbers your way if you're interested.

--titus
That's awesome, thanks for checking this out Titus!

-chris
cjfields is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 10:22 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO