Filtering Illumina data to reduce file size

[Mona]

Hello,
I have paired end data from Illumina hi seq for a bacterial genome that has been sequenced using three different insert sizes, 160, 305 and 505 respectively. My task is to perform de novo assembly of the genome but the problem is that every single file contains more than 60 million reads and its not possible to run assembly of this much large file. Is there any way I can reduce the size of the file, by removing some reads?? or performing some kind of filteration?

[westerman]

Look for Titus Brown's 'diginorm' program. It does an intelligent reduction of data. Seems to work well for genomic data. Perhaps not so well for transcriptome data.

[Mona]

Thanks for the suggestion, i will try that and get back for further problems

[nickloman]

Subsample reads from your files using Heng Li's seqtk program (https://github.com/lh3/seqtk) and the "sample" command.

[westerman]

Subsample reads from your files using Heng Li's seqtk program (https://github.com/lh3/seqtk) and the "sample" command.

If you are just going to randomly throw away reads then you might as well go the cheap route and not do as much sequencing in the first place. No disrespect to Li's program but since diginorm provides for an intelligent reduction of reads then I suggest using it instead of a random selection.

[krobison]

At the Boston Illumina User's Group meeting today, Illumina mentioned that BaseSpace will have an option for "quality-binning" -- by reducing quality scores to a small number of bins, the data compresses quite a bit (they claimed 50% reduction in compressed FASTQ size). An underlying assumption is that quality scores offer more gradation than programs really find useful.

Pretty trivial to implement in Perl, though I leave that as an exercise for the student :-)