Filtering SAM files by base quality (QUAL)?

[JPreston]

Hello,

I need to filter my SAM/BAM files to remove bases below a certain base quality (QUAL) threshold, eg. Q30. Does anybody know if there any tools for this available?

Thanks in advance!

[Brian Bushnell]

BBTools can discard reads from a sam file with average quality below a threshold:

reformat.sh in=reads.sam out=filtered.sam maq=30

However, pairing would get messed up, if the reads are paired. It's easier to do quality-related stuff on the fastq file, then map.

[JPreston]

OK thanks a lot! Do you know of any programs that can mask fastq bases by quality, rather than discarding the read?

[Brian Bushnell]

The same tool can trim low-quality bases, rather than discarding reads:

reformat.sh in=reads.fastq out=trimmed.fastq qtrim=rl trimq=30

(include in2 and out2 if you have paired reads in 2 files)

That command would trim the right and left sides of each read with a quality threshold of 30. Masking would be less useful, normally. And by the way, 30 is typically way too high of a threshold for trimming; I recommend around 10 for most uses.

[JPreston]

Thanks again. I need a way to keep all of the bases above the quality threshold, rather than trimming the ends. I also want the quality scores to be very high for this application. It should be straightforward to write a script to filter and premask a fastq file, but I was hoping there was already a tool for this available. I wasn't sure how to deal with the quality scores of the masked bases.

[Brian Bushnell]

I'm not sure what you could do with fastq reads that have all bases with quality below some threshold masked... so I have not written anything to do that. What are you trying to accomplish?

[JPreston]

I am working on a method to call low-frequency SNPs in heterogeneous populations. The method is based on using overlapping paired-end reads to sequence each DNA molecule twice. This eliminates sequencing errors because an error will only occur in one of the reads. We use a program called SeqPrep to overlap the reads and generate a consensus sequence with very high quality. I have been using a program called LoFreq to call these low frequency SNPs but there is no way to filter the calls based on quality. I want to filter my reads by base quality in order to see how the various methods compare at various quality thresholds.

[Brian Bushnell]

reformat.sh works with paired reads, it's just that the sam/bam format does not ensure paired reads are together, so it's possible for a read to be filtered out and its mate to be kept, which may or may not matter, depending on the application.

I also have a program called "bbmerge" which will merge overlapped reads to create a consensus:

bbmerge.sh in=read1.fq in2=read2.fq out=merged.fq outu=unmerged.fq

It's very fast and much more accurate than another commonly used tool, Flash (I have not tried SeqPrep, though). The output (merged.fq) is no longer paired, because the merged reads only create 1 consensus sequence, so it seems odd to output two copies of it. So the bam file would be filled with single-ended reads anyway, at least for the reads that were successfully merged.

[GenoMax]

It is odd that you are not doing your quality filtering up front. With SeqPrep you should be able to do that as you overlap the reads.

[JPreston]

I do some quality filtering before mapping but I know of no program that will mask the bases rather than trim or remove the reads completely. I don't want to trim reads at such a high quality threshold because I will lose too many bases.
Also, I don't know of an option in seqprep that works to quality filter the output.

[yueluo]

You can specify your threshold for base-trimming.
Anyway, what do you mean by masking bases?

[JPreston]

By masking I mean replacing low quality bases with an N.

[JPreston]

I used BBmap to trim the reads. It worked well. Thanks!

[Brian Bushnell]

I used BBmap to trim the reads. It worked well. Thanks!

You're welcome!

[MNMoller]

Dear JPreston

Did you find a solution to your problem. I have the same issue and is looking for a tool to filter out bases within a read with low quality value. The tool should be able to do this on the SAM file, or on the FASTQ file, as long as it wont affect the alignment; for example by replacing it with an N like you suggest.