duplicate reads removal

vasvale · 09-17-2009, 06:43 PM

is there any software that removes duplicate single-reads? (Casava does it for paired-end reads only)

krobison · 09-18-2009, 05:28 AM

According to the man page, SAMTools has a mode to do this

http://samtools.sourceforge.net/samtools.shtml

mdup samtools rmdup <input.srt.bam> <out.bam>
Remove potential PCR duplicates: if multiple read pairs have identical external coordinates, only retain the pair with highest mapping quality. This command ONLY works with FR orientation and requires ISIZE is correctly set.

rmdupse samtools rmdupse <input.srt.bam> <out.bam>
Remove potential duplicates for single-ended reads. This command will treat all reads as single-ended even if they are paired in fact.

lh3 · 09-18-2009, 05:43 AM

For duplicate removal, Picard is recommended. It does a better job than samtools-C.

Nomijill · 02-03-2010, 03:48 AM

Hi,

I am educating myself on duplicate removal. Why/How is Picard better than Samtools?

Thanks.

lh3 · 02-03-2010, 06:13 AM

Picard removes duplicates across chromosomes, but samtools cannot.

JohnK · 10-29-2010, 12:01 PM

Quote:

Originally Posted by lh3

Picard removes duplicates across chromosomes, but samtools cannot.

Is that the only notable difference?

corthay · 11-09-2010, 11:06 PM

Hi,

Is there any software that removes duplicate of PE or MP read
before mapping ? I would like to remove duplicate before doing
de novo assembly.
Thanks.

Corthay

drio · 11-10-2010, 05:11 AM

There is no way to determine what is a PCR duplicate at that level. That is why it has to be done at mapping level. Even then, not all of them are true PCR duplicates (read lh3's statistical calculation of the expected number of PCR dups to find in a sample).

lh3 · 11-10-2010, 06:04 AM

It is possible to dedup before mapping. You may hash the first 14bp of each end and discard a pair if the 14+14bp coincides another pair. This method is not as good as deduping after mapping, but should be good enough. On the other hand, I do not think deduping is quite necessary for assembly.

corthay · 11-11-2010, 03:25 AM

Quote:

Originally Posted by lh3

It is possible to dedup before mapping. You may hash the first 14bp of each end and discard a pair if the 14+14bp coincides another pair. This method is not as good as deduping after mapping, but should be good enough. On the other hand, I do not think deduping is quite necessary for assembly.

Thanks for the idea. I just would like to check if deduping is necessary for assembly as Panda Genome paper did it for long insert-sizes libraries.

Corthay

JohnK · 11-11-2010, 05:14 PM

Quote:

Originally Posted by corthay

Thanks for the idea. I just would like to check if deduping is necessary for assembly as Panda Genome paper did it for long insert-sizes libraries.

Corthay

Hi, Corthay.

I remove duplicates for SE and PE stuffs always. PE you should be removing between 5 and 15 percent, and for SE it'll be significantly larger and anywhere between 30 to even possibly 60 percent of your reads. It depends on the quality of the PCR step of course, which I personally know little about. Also, removing duplicates really only depends on what you're doing. If you're looking at ngs/mps/hts stuffs and you wish to accurately determine all the SNPs in your data, you probably don't have time to go through each variant that's called and so you want the most accurate call. You'd remove the duplicates. However, if you have a single gene of interest you can just as easily visually inspect whatever region or SNP, regardless of whether you removed the duplicates, and determine whether that 'call' is valid or not.

ikrier · 03-03-2011, 02:21 AM

I have tried to use the rmdup command and have found something quite strange.

I have a sam file from my alignment. I view it as a bam, and then filter on quality with :
/data/common/programs/samtools/samtools view -h $f.srt.bam | awk '{if($5 >= 10 || $1 == "@SQ" || $1 == "@PG") print $0}' | /data/common/programs/samtools/samtools view -bS - > $f.srt.unique-qual-ge10.bam

this gives me the file I want to work with. I need an output for quest, with duplicates removed, so what I tried was :
1. First get the fields in the format needed for quest then use the UNIX sort command to get the alignments with unique chromosome, position and strand.
2. First use rmdup to get a new bam file then get the fields in the format needed for quest

And the two results are different. I would have assumed that rmdup would remove the alignments with the same chromosome, strand and position, so that if I extract sequences with sort -u for these fields I would find the same number in the end.

Can anyone explain this?

ikrier · 03-03-2011, 08:56 AM

We looked into it in the end, and it simply turns out that reads with an insertion/deletion in the alignment get their start position shifted in the output, but samtools rmdup takes it into account when removing the PCR duplicates.

I have definitely learned something today.

ttnguyen · 03-09-2011, 09:17 AM

What is acceptable PCR duplicate percentage in a ChIP-seq dataset and in a RNA-seq dataset after mapping?

In my ChIP-seq dataset, after mapping I found 66% duplicate by using Picard. I think this is too high so wanna know what is acceptable duplicate level?

JohnK · 03-09-2011, 10:04 AM

Quote:

Originally Posted by ttnguyen

What is acceptable PCR duplicate percentage in a ChIP-seq dataset and in a RNA-seq dataset after mapping?

In my ChIP-seq dataset, after mapping I found 66% duplicate by using Picard. I think this is too high so wanna know what is acceptable duplicate level?

It's not too high necessarily. It really depends on starting DNA quantity and how much you PCR it up. I've seen between 30% and even up to over 80% depending on the context of the protein we're after.

husamia · 07-01-2011, 01:56 PM

I have seen duplicate percentage of 33% in targeted high coverage experiment and based on my assessment of false positive and false negative I am happy not removing duplicates.

Michael.James.Clark · 07-01-2011, 07:07 PM

Quote:

Originally Posted by husamia

I have seen duplicate percentage of 33% in targeted high coverage experiment and based on my assessment of false positive and false negative I am happy not removing duplicates.

It depends on what your experiment is. In some cases it has very little effect, in some it can be detrimental. In no case that I've heard of does leaving PCR duplicates in your data improve your results, however.

panos_ed · 01-07-2015, 03:15 AM

Quote:

Originally Posted by lh3

It is possible to dedup before mapping. You may hash the first 14bp of each end and discard a pair if the 14+14bp coincides another pair. This method is not as good as deduping after mapping, but should be good enough. On the other hand, I do not think deduping is quite necessary for assembly.

Why 14bp? Why not something else?

transforu · 01-07-2015, 08:07 PM

Hi. everybody
I have a problem about remove duplicates,
In this study , http://www.ncbi.nlm.nih.gov/Traces/sra/?study=ERP000603
It's have 2 Experiments, and 13 RUNs,
How to do remove duplicates? by study? by Experiments? or by one runs?
Thanks !!

sarvidsson · 01-08-2015, 01:59 AM

Remove PCR duplicates by library (and this looks like one big "library" mixed from 4 PCR pools). Optical duplicates are removed by lane (but I guess Picard would detect the within lane reads, given proper read group labelling).

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
removal of unpaired reads	bioenvisage	Bioinformatics	14	08-08-2014 06:30 AM
Duplicate Reads	myronpeto	Bioinformatics	7	03-07-2013 08:36 AM
PCR duplicate removal for whole genome sequencing vs. whole exome sequencing	cliff	Bioinformatics	1	09-27-2011 08:29 AM
Removal of poor quality reads before alignment	gibsongenetics	Bioinformatics	2	05-16-2011 06:22 AM
threshold for duplicate removal?	mard	Bioinformatics	2	03-21-2010 04:45 PM

09-17-2009, 06:43 PM	#1
vasvale Member Location: Seattle Join Date: Mar 2008 Posts: 29	duplicate reads removal is there any software that removes duplicate single-reads? (Casava does it for paired-end reads only)

02-03-2010, 03:48 AM	#4
Nomijill Member Location: Southwest Florida Join Date: Sep 2009 Posts: 24	Duplicate removal Hi, I am educating myself on duplicate removal. Why/How is Picard better than Samtools? Thanks.

11-09-2010, 11:06 PM	#7
corthay Member Location: japan Join Date: Oct 2008 Posts: 25	Removing duplicates before mapping. Hi, Is there any software that removes duplicate of PE or MP read before mapping ? I would like to remove duplicate before doing de novo assembly. Thanks. Corthay

11-10-2010, 05:11 AM	#8
drio Senior Member Location: 41°17'49"N / 2°4'42"E Join Date: Oct 2008 Posts: 323	There is no way to determine what is a PCR duplicate at that level. That is why it has to be done at mapping level. Even then, not all of them are true PCR duplicates (read lh3's statistical calculation of the expected number of PCR dups to find in a sample). __________________ -drd

09-18-2009, 05:28 AM	#2
krobison Senior Member Location: Boston area Join Date: Nov 2007 Posts: 747	According to the man page, SAMTools has a mode to do this http://samtools.sourceforge.net/samtools.shtml mdup samtools rmdup <input.srt.bam> <out.bam> Remove potential PCR duplicates: if multiple read pairs have identical external coordinates, only retain the pair with highest mapping quality. This command ONLY works with FR orientation and requires ISIZE is correctly set. rmdupse samtools rmdupse <input.srt.bam> <out.bam> Remove potential duplicates for single-ended reads. This command will treat all reads as single-ended even if they are paired in fact.

09-18-2009, 05:43 AM	#3
lh3 Senior Member Location: Boston Join Date: Feb 2008 Posts: 693	For duplicate removal, Picard is recommended. It does a better job than samtools-C.

02-03-2010, 06:13 AM	#5
lh3 Senior Member Location: Boston Join Date: Feb 2008 Posts: 693	Picard removes duplicates across chromosomes, but samtools cannot.

11-10-2010, 06:04 AM	#9
lh3 Senior Member Location: Boston Join Date: Feb 2008 Posts: 693	It is possible to dedup before mapping. You may hash the first 14bp of each end and discard a pair if the 14+14bp coincides another pair. This method is not as good as deduping after mapping, but should be good enough. On the other hand, I do not think deduping is quite necessary for assembly.

03-03-2011, 02:21 AM	#12
ikrier Member Location: Lausanne Join Date: Dec 2009 Posts: 19	I have tried to use the rmdup command and have found something quite strange. I have a sam file from my alignment. I view it as a bam, and then filter on quality with : /data/common/programs/samtools/samtools view -h $f.srt.bam \| awk '{if($5 >= 10 \|\| $1 == "@SQ" \|\| $1 == "@PG") print $0}' \| /data/common/programs/samtools/samtools view -bS - > $f.srt.unique-qual-ge10.bam this gives me the file I want to work with. I need an output for quest, with duplicates removed, so what I tried was : 1. First get the fields in the format needed for quest then use the UNIX sort command to get the alignments with unique chromosome, position and strand. 2. First use rmdup to get a new bam file then get the fields in the format needed for quest And the two results are different. I would have assumed that rmdup would remove the alignments with the same chromosome, strand and position, so that if I extract sequences with sort -u for these fields I would find the same number in the end. Can anyone explain this?

03-03-2011, 08:56 AM	#13
ikrier Member Location: Lausanne Join Date: Dec 2009 Posts: 19	We looked into it in the end, and it simply turns out that reads with an insertion/deletion in the alignment get their start position shifted in the output, but samtools rmdup takes it into account when removing the PCR duplicates. I have definitely learned something today.

03-09-2011, 09:17 AM	#14
ttnguyen Member Location: Ireland Join Date: Mar 2010 Posts: 41	What is acceptable PCR duplicate percentage in a ChIP-seq dataset and in a RNA-seq dataset after mapping? In my ChIP-seq dataset, after mapping I found 66% duplicate by using Picard. I think this is too high so wanna know what is acceptable duplicate level?

07-01-2011, 01:56 PM	#16
husamia Member Location: cinci Join Date: Apr 2010 Posts: 66	I have seen duplicate percentage of 33% in targeted high coverage experiment and based on my assessment of false positive and false negative I am happy not removing duplicates.

01-07-2015, 08:07 PM	#19
transforu Junior Member Location: Taiwan Join Date: Sep 2014 Posts: 5	Hi. everybody I have a problem about remove duplicates, In this study , http://www.ncbi.nlm.nih.gov/Traces/sra/?study=ERP000603 It's have 2 Experiments, and 13 RUNs, How to do remove duplicates? by study? by Experiments? or by one runs? Thanks !!

01-08-2015, 01:59 AM	#20
sarvidsson Senior Member Location: Berlin, Germany Join Date: Jan 2015 Posts: 137	Remove PCR duplicates by library (and this looks like one big "library" mixed from 4 PCR pools). Optical duplicates are removed by lane (but I guess Picard would detect the within lane reads, given proper read group labelling).