Picard failure on NextSeq data

[TonyBrooks]

So, we've just begun to run some of our new NextSeq data through our standard pipelines and hit a snag. bcl2fastq still gives us fastq by index, read and lane, so the eight files for every sample are zcat'd together into two and aligned using either STAR or tophat. The problems begin to happen when we use picard Mark Duplicates which seems to fail on certain samples.

Exception in thread "main" htsjdk.samtools.SAMException: Value was put into PairInfoMap more than once.  1: null:NS500195:9:H0MT4AGXX:1:11201:10896:15588
	at htsjdk.samtools.CoordinateSortedPairInfoMap.ensureSequenceLoaded(CoordinateSortedPairInfoMap.java:132)
	at htsjdk.samtools.CoordinateSortedPairInfoMap.remove(CoordinateSortedPairInfoMap.java:86)
	at picard.sam.DiskReadEndsMap.remove(DiskReadEndsMap.java:63)
	at picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:434)
	at picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:177)
	at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:183)
	at picard.sam.MarkDuplicates.main(MarkDuplicates.java:161)

I've been trying to diagnose the problem and I think it may be related to samples having clusters with reads at the same XY co-ordinate, albeit on different lanes - although we've not had this problem on HiSeq rapid runs.
If I rename the fastq headers to a rolling number using fastx_renamer -n COUNT and this "fixes" the problem although I now no longer have any info about optical duplicates.
Has anyone else hit this problem?

[mgogol]

I'm just starting out with NextSeq data by installing bcl2fastq version 2. Haven't gotten as far as all that yet, but thanks for the heads up.

[TonyBrooks]

I've spoken to Illumina tech support and they have pointed out a possible bug in bcl2fastq v2.

"We identified a bug caused by thread handling that in certain cases caused errors in the order of reads such that R1 and R2 files were not in the same order. This is intermittent and may fit with your experience since it depends purely on the thread memory handling which can differ from run to run."

I've re-created the fastq using a single thread. It obviously takes 8 times as long, but I'm not seeing the picard error this time. They've promised a new version with the bug fixed any day now, so keep your eyes open.

[cnicolet]

nextseq bcl2fastq

this is a little off topic from the thread but seems related to some of the things both you guys are doing--do you have a slick way to zcat the bcl2fastq output correctly? I've been doing it "by hand" two files at a time

[TonyBrooks]

this is a little off topic from the thread but seems related to some of the things both you guys are doing--do you have a slick way to zcat the bcl2fastq output correctly? I've been doing it "by hand" two files at a time

I'm using a modified version of a script from Shingo Kikugawa
It's single thread, so takes a while. It's still quicker than manual zcatting.

I spoke to Illumina about this and a version of bcl2fastq is in the pipeline that will have a flag to stop lane splitting of fastq.