Exact duplicate reads/readnames/quality/tiles in Novaseq FASTQs

[ECO]

Anyone seen this? Some (not correlated with software versions AFAICT) of our MarkDuplicates jobs are failing because there are exact copies of the same reads in the fastq. Identical readnames, locations/tiles, sequence, qualities.

Really weird...

[GenoMax]

Sounds like bcl2fastq experienced a software issue. I suggest that you re-run the demultiplexing. I have seen this posted rarely and if I recall had experienced it one time. bcl2fastq re-run fixed the problem.

I will also put a plug in for clumpify.sh from BBMap suite. It allows detection of all/optical dups without alignment of data.

[ECO]

We've run it several times to no effect, but are re-running it now with single threaded writing...I'm hopeful that helps....

[GenoMax]

Interesting. Let us know what happens.

[saimonara]

I'm experiencing this problem as well! Unfortunately, I just notice this error and a third of the ~40 runs we've done have the problem. The runs are not sequential and there were no software updates or changes to the scripts that start the demultiplexing. I'm using bcl2fastq v2.19.1.403.
Once I run bcl2fastq again with the same original settings (`bcl2fastq -r 4 -p 4 -w 4`) , the process doesn't generate duplicated data.
Has anyone found a way to fix this? What's going on?!

[kcchan]

Have you tried upgrading bcl2fastq v2.20? v2.19 was really buggy for NovaSeq data

[saimonara]

Hi kcchan,

yes, I updated to v2.20 this morning but saw the same behavior later in the day. However, when I reran bcl2fastq I got the correct number of reads. I think it might be something that my pipeline is doing.