seq_crumbs memory issues

[DrWorm]

Hey all,

I've been loving the seq_crumbs package for trimming and filtering my Illumina data, and was excited to download the latest update that can handle paired end data. Unfortunately, when I tried to use the tools I noticed that the memory requirements had increased dramatically. I expected a modest increase in processing paired end data compared to single end files, but it was jumping from 8G for single end to 80G for pe. Has anyone else encountered this issue or am I doing something totally wrong?

Also, does anyone else know of tools for quality filtering and complexity filtering that maintain read pairs? I'm aware of trimmomatic and trim_galore for quality trimming, but I didn't think they do filtering based on the quality/complexity of the read overall.

Thanks!

[peio]

Originally Posted by DrWorm

Hey all,

I've been loving the seq_crumbs package for trimming and filtering my Illumina data, and was excited to download the latest update that can handle paired end data. Unfortunately, when I tried to use the tools I noticed that the memory requirements had increased dramatically. I expected a modest increase in processing paired end data compared to single end files, but it was jumping from 8G for single end to 80G for pe. Has anyone else encountered this issue or am I doing something totally wrong?

Also, does anyone else know of tools for quality filtering and complexity filtering that maintain read pairs? I'm aware of trimmomatic and trim_galore for quality trimming, but I didn't think they do filtering based on the quality/complexity of the read overall.

Thanks!

Hi DrWorm,

I am one of the developers of the seq_crumbs package. We're interested in the memory hogging issue.
Versions prior to 0.1.8 had memory related bug that we solved in the 0.1.8 version.

seq_crumb uses python's standar library for multiprocessing and seems that versions of python <=2.7.3 are affected by a bug related with memory consumption.

With python 2.7.4 in the latest ubuntu the memory use stays stable but with latest debian and python 2.7.3 memory use grows untill the process is finished.

Which version of python are you using?
p.

[DrWorm]

I believe we're using Python 2.6.5. I'll request an update and see if that makes a difference.

Thanks for the advice!!