Exact duplicate reads/readnames/quality/tiles in Novaseq FASTQs

ECO · 02-28-2019, 08:21 AM

Anyone seen this? Some (not correlated with software versions AFAICT) of our MarkDuplicates jobs are failing because there are exact copies of the same reads in the fastq. Identical readnames, locations/tiles, sequence, qualities.

Really weird...

GenoMax · 02-28-2019, 08:27 AM

Sounds like bcl2fastq experienced a software issue. I suggest that you re-run the demultiplexing. I have seen this posted rarely and if I recall had experienced it one time. bcl2fastq re-run fixed the problem.

I will also put a plug in for clumpify.sh from BBMap suite. It allows detection of all/optical dups without alignment of data.

ECO · 02-28-2019, 12:07 PM

We've run it several times to no effect, but are re-running it now with single threaded writing...I'm hopeful that helps....

GenoMax · 02-28-2019, 03:25 PM

Interesting. Let us know what happens.

saimonara · 12-05-2019, 10:44 AM

I'm experiencing this problem as well! Unfortunately, I just notice this error and a third of the ~40 runs we've done have the problem. The runs are not sequential and there were no software updates or changes to the scripts that start the demultiplexing. I'm using bcl2fastq v2.19.1.403.
Once I run bcl2fastq again with the same original settings (`bcl2fastq -r 4 -p 4 -w 4`) , the process doesn't generate duplicated data.
Has anyone found a way to fix this? What's going on?!

kcchan · 12-05-2019, 02:50 PM

Have you tried upgrading bcl2fastq v2.20? v2.19 was really buggy for NovaSeq data

saimonara · 12-05-2019, 02:53 PM

Hi kcchan,

yes, I updated to v2.20 this morning but saw the same behavior later in the day. However, when I reran bcl2fastq I got the correct number of reads. I think it might be something that my pipeline is doing.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Variable per tile sequence quality across amplicon-seq fastqs	dross11	Illumina/Solexa	2	06-15-2017 12:58 AM
RIPSeeker "Error in NSBS(i, x, exact = exact, upperBoundIsStrict = !allow.append) : .	sbcn	Bioinformatics	0	01-31-2017 09:58 AM
Sequence Quality for large number of fastqs	bio_informatics	Bioinformatics	14	11-12-2015 10:55 AM
Trimming reads messes up PE fastqs? any ideas please?	swNGS	Bioinformatics	24	07-26-2013 03:02 PM
dwgsim -> readnames and quality scores	genome	Bioinformatics	1	02-16-2011 07:17 PM

02-28-2019, 08:21 AM	#1
ECO --Site Admin-- Location: SF Bay Area, CA, USA Join Date: Oct 2007 Posts: 1,358	Exact duplicate reads/readnames/quality/tiles in Novaseq FASTQs Anyone seen this? Some (not correlated with software versions AFAICT) of our MarkDuplicates jobs are failing because there are exact copies of the same reads in the fastq. Identical readnames, locations/tiles, sequence, qualities. Really weird...

02-28-2019, 08:27 AM	#2
GenoMax Senior Member Location: East Coast USA Join Date: Feb 2008 Posts: 6,992	Sounds like bcl2fastq experienced a software issue. I suggest that you re-run the demultiplexing. I have seen this posted rarely and if I recall had experienced it one time. bcl2fastq re-run fixed the problem. I will also put a plug in for clumpify.sh from BBMap suite. It allows detection of all/optical dups without alignment of data.

02-28-2019, 12:07 PM	#3
ECO --Site Admin-- Location: SF Bay Area, CA, USA Join Date: Oct 2007 Posts: 1,358	We've run it several times to no effect, but are re-running it now with single threaded writing...I'm hopeful that helps....

02-28-2019, 03:25 PM	#4
GenoMax Senior Member Location: East Coast USA Join Date: Feb 2008 Posts: 6,992	Interesting. Let us know what happens.

12-05-2019, 10:44 AM	#5
saimonara Junior Member Location: Bay Area, CA, USA Join Date: Feb 2014 Posts: 7	I'm experiencing this problem as well! Unfortunately, I just notice this error and a third of the ~40 runs we've done have the problem. The runs are not sequential and there were no software updates or changes to the scripts that start the demultiplexing. I'm using bcl2fastq v2.19.1.403. Once I run bcl2fastq again with the same original settings (`bcl2fastq -r 4 -p 4 -w 4`) , the process doesn't generate duplicated data. Has anyone found a way to fix this? What's going on?!

12-05-2019, 02:50 PM	#6
kcchan Senior Member Location: USA Join Date: Jul 2012 Posts: 183	Have you tried upgrading bcl2fastq v2.20? v2.19 was really buggy for NovaSeq data

12-05-2019, 02:53 PM	#7
saimonara Junior Member Location: Bay Area, CA, USA Join Date: Feb 2014 Posts: 7	Hi kcchan, yes, I updated to v2.20 this morning but saw the same behavior later in the day. However, when I reran bcl2fastq I got the correct number of reads. I think it might be something that my pipeline is doing.