Data preprocessing

Katerina Cvikova · 06-07-2017, 01:13 AM

Hi,
do you use a some rule for trimming/removing or saving reads with the worst quality for DNA and RNA assembly/alignment. How do you decide to trimm/remove/save? According the type of analysis, coverage, base quality.... ???? Thanks a lot.

GenoMax · 06-07-2017, 04:49 AM

In general you can do without explicit quality filtering, unless you know that you data contains a significant portion of low quality reads.

If you are aligning to a known reference you can get away with using data that may be as low as Q10. If you are doing de novo work then that would likely be the only case where quality score based filtering is warranted. You would want to filter out data Q25 or lower for that application.

Dario1984 · 06-27-2017, 11:00 PM

It's always best to do adapter trimming, even if the aligner you use later can do soft-clipping, such as STAR. It would work moderately faster if it didn't have to attempt to map the adapter sequences to the genome and then soft-clip the sequences when no alignment was found. You can have a look at the overall quality profile of the reads in a sample using a tool like FastQC. It's worth applying quality filtering, even if very few reads in the dataset are removed by it. There could be a rare circumstance where it makes an important difference.

deep639 · 06-28-2017, 09:48 AM

Its always good to trim the adapters and do quality trimming before running alignment. Quality trimming is usually done at q20 level. Programs like trim_galore can auto detect the adapter sequence that needs to be trimmed based on the input reads.

Brian Bushnell · 06-28-2017, 10:45 AM

Quote:

Originally Posted by deep639

Its always good to trim the adapters and do quality trimming before running alignment. Quality trimming is usually done at q20 level. Programs like trim_galore can auto detect the adapter sequence that needs to be trimmed based on the input reads.

I disagree with the part about quality-trimming. There is at least one published study indicating trimming to high levels like Q20 is generally detrimental to alignment, which agrees with my observations. However, it depends on the aligner. For something like bowtie1 which can only tolerate 3 mismatches, long reads >100bp might indeed need very aggressive quality-trimming... but then, you shouldn't use bowtie1 with long reads.

I generally suggest the range of Q5-Q12 for quality-trimming using HiSeq/MiSeq Illumina data that has the full range of quality scores. Illumina is moving toward binned and inaccurate Q-scores on its latest platforms, though, so the utility of quality-trimming is going to be reduced.

deep639 · 06-28-2017, 10:51 AM

Quote:

Originally Posted by Brian Bushnell

I disagree with the part about quality-trimming. There is at least one published study indicating trimming to high levels like Q20 is generally detrimental to alignment, which agrees with my observations. However, it depends on the aligner. For something like bowtie1 which can only tolerate 3 mismatches, long reads >100bp might indeed need very aggressive quality-trimming... but then, you shouldn't use bowtie1 with long reads.

I generally suggest the range of Q5-Q12 for quality-trimming using HiSeq/MiSeq Illumina data that has the full range of quality scores. Illumina is moving toward binned and inaccurate Q-scores on its latest platforms, though, so the utility of quality-trimming is going to be reduced.

I think trim_galore by default sets it quality trimming to 20, I have never tried to change the setting, usually a small percentage of reads are thrown out because of quality and adapter trimming.

Brian Bushnell · 06-28-2017, 11:01 AM

The problem is not so much the fraction of data that is discarded, but rather, the bias - Illumina read quality is affected by sequence content, so a high quality-trimming or quality-filtering thresholds can disparately impact certain genomic regions. This is particularly important for quantitative analyses. And regardless of bias, longer reads give more accurate mapping. The confidence of a 250bp alignment, and the ability to place it correctly despite inexact repeats in the genome, is much higher compared to a 150bp read, even if the last 100bp of the 250bp read are only Q17 and thus would be expected to contain 2 mismatches. For variant-calling, q-trimming can be done AFTER alignment to allow the most accurate mapping possible.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Preprocessing before k-means clustering of expression data	Mchicken	Bioinformatics	1	04-26-2016 01:30 AM
If I delete one sample do I have to redo DNA methylation data preprocessing	HeidiLee	Bioinformatics	1	05-08-2014 10:28 AM
Preprocessing needed for RNA-Seq data	PFS	Bioinformatics	10	03-06-2014 09:36 AM
Test for RNAseq data preprocessing step (with regards to adapter and hexamer)	lynnco2008	RNA Sequencing	5	10-01-2012 09:02 AM
RNA seq data preprocessing and cleaning	figo1019	RNA Sequencing	0	07-17-2012 01:45 AM

06-07-2017, 01:13 AM	#1
Katerina Cvikova Junior Member Location: Olomouc Join Date: Apr 2014 Posts: 6	Data preprocessing Hi, do you use a some rule for trimming/removing or saving reads with the worst quality for DNA and RNA assembly/alignment. How do you decide to trimm/remove/save? According the type of analysis, coverage, base quality.... ???? Thanks a lot.

06-07-2017, 04:49 AM	#2
GenoMax Senior Member Location: East Coast USA Join Date: Feb 2008 Posts: 7,080	In general you can do without explicit quality filtering, unless you know that you data contains a significant portion of low quality reads. If you are aligning to a known reference you can get away with using data that may be as low as Q10. If you are doing de novo work then that would likely be the only case where quality score based filtering is warranted. You would want to filter out data Q25 or lower for that application.

06-27-2017, 11:00 PM	#3
Dario1984 Senior Member Location: Sydney, Australia Join Date: Jun 2011 Posts: 166	It's always best to do adapter trimming, even if the aligner you use later can do soft-clipping, such as STAR. It would work moderately faster if it didn't have to attempt to map the adapter sequences to the genome and then soft-clip the sequences when no alignment was found. You can have a look at the overall quality profile of the reads in a sample using a tool like FastQC. It's worth applying quality filtering, even if very few reads in the dataset are removed by it. There could be a rare circumstance where it makes an important difference.

06-28-2017, 09:48 AM	#4
deep639 Member Location: Irvine Join Date: Dec 2013 Posts: 10	Its always good to trim the adapters and do quality trimming before running alignment. Quality trimming is usually done at q20 level. Programs like trim_galore can auto detect the adapter sequence that needs to be trimmed based on the input reads.

06-28-2017, 11:01 AM	#7
Brian Bushnell Super Moderator Location: Walnut Creek, CA Join Date: Jan 2014 Posts: 2,707	The problem is not so much the fraction of data that is discarded, but rather, the bias - Illumina read quality is affected by sequence content, so a high quality-trimming or quality-filtering thresholds can disparately impact certain genomic regions. This is particularly important for quantitative analyses. And regardless of bias, longer reads give more accurate mapping. The confidence of a 250bp alignment, and the ability to place it correctly despite inexact repeats in the genome, is much higher compared to a 150bp read, even if the last 100bp of the 250bp read are only Q17 and thus would be expected to contain 2 mismatches. For variant-calling, q-trimming can be done AFTER alignment to allow the most accurate mapping possible.