[nickloman]

That is, it looks to me from the chart that the MiSeq bin of bases with a quality score of 20 (1 error in 100) actually had 1 error in 1000. Indicating that they really were Q30 bases, rather than Q20 bases?

That's right, but bear in mind there were very few bases with that quality in the dataset (c.f. top panel of Figure 1). I tried to represent this with the area of the point, but I'm not sure I did a very good job getting the dynamic range across.

[ramujana]

Thank you all for the information. It seems i'm leaning towards MiSeq.

[MrGuy]

Then be sure to go in with your eyes wide open. This came out today:

http://www.genomeweb.com/sequencing/rising-demand-illuminas-sequencing-reagents-causes-shipment-delays-customer-frus

[pmiguel]

Then be sure to go in with your eyes wide open. This came out today:
http://www.genomeweb.com/sequencing/...-customer-frus

Okay, but I have seen issues of this sort from LifeTech/ABI, Roche and Illumina at various points.

I think it is probably the result of commonly used book-keeping (inventory control) used at many companies. Basically they can see a loss if they have to throw out or store some reagent. Whereas they do a poor job of capturing the loss in sales due to instruments sitting idle waiting for reagents. So they optimize on the parameter they measure and ignore the one that they don't measure.

--
Phillip

[Elcannibal]

Back order issues are certainly not unique to Illumina, in fact they probably stand out because their customers are less used to it !

[pmiguel]

Back order issues are certainly not unique to Illumina, in fact they probably stand out because their customers are less used to it !

Let's not go overboard here. Illumina does have perennial issues with back orders. Once or twice, you can attribute to scaling issues, etc. But intermittent failures to have enough stock on hand to quickly process orders that stretches out over years points to design flaws in their supply chain.

Applied Biosystems frequently had issues of this sort in the SOLiD1-4 days. But often they were of the form one item out of 30 needed to make libraries and do a run would be not available.

The poor instrument-run assessment/trouble-shooting tools on the 454 makes it nearly impossible to determine whether you got bad reagents from Roche, or your runs are working sub-par for some other reason. But in my paranoid moments I suspect they shipped a lot of marginal lots of reagents.

Anyway, I think the message is that all the companies suck in this regard. Could be an area where improvement would give them a competitive advantage, but either they can't see that or see the issues they face as intractable.

--
Phillip

[mattarno]

instrument reliability

sorry to stick my oar in but I'd like to ask the forum a (perhaps naive) question regarding reliability of illumina sequencers. speaking from anecdotal experience (i.e. chatting to various illumina owners in and around london) it seems that there are lots of HiSeq/HiScan machines that have broken down since installation and required extensive repair/rebuilding and even complete replacement.

So my questions are: how many other illumina owners have experienced reliability problems, including significant down time and machine replacement? Is this just isolated instances or my bias? is it possible to assess the scale of the problem (i.e. % of machines sold, or % of potential run time lost due to repairs etc.) if it actually exists?

if this is a real issue, is there an official line or response from Illumina?

how do illumina machines compare with other platforms (ion torrent, roche/454) for reliability - bearing in mind the illumina market share?

thanks for any input on these questions, I would really welcome the opinions of the seqanswers community.

Matt

[Nitrogen-DNE-sulfer]

Its a good paper and I'll throw my unsolicited recommendations on there as well. I think Nick was generous with the Ion platform. The Library prep, EmPCR and CAFIE correction CPU time after the run were not baked into the time comparison. Putting Ion down as a 3 hr run should really be more like 3hr of Lib+emPCR+enrichment+quant up front, then a 3hr run, and then a few hours of CPU to get CAFIE adjusted fastqs before you have reads you can run with. Nextera + MiSeq run should be the apples to apples time comparison and this should be normalized to read length. 300 cycles of terminator data next to 100bp of homopolymer prone extentions where on average 2.3 bases per cycle are acquired is not apples to apples. In light of this, I dont see a single thing the Ion is good at (at this time point).
I'm also seeing that the Ion data only used 90% of the data while the ILMN and 454 were using closer to 99% of the data. If the Ion data was filtered more aggressively and still had worse performance its a bit unsettling.. Is this the right way to read this are did the emulsions have higher doublets on the poisson dilution than the 454 data?

Ion is certainly improving quickly but I see 454 as a prophet for their future. 7 years later the homopolymer issues is still only 5X better and not enough to displace the indel error significantly found in the supp. 1000X higher indel error rate for ion is buried in the supplement and the Ion data appears to have an additional stranded bias the 454 platform doesnt have. My advice is to not pay too much attention to the futures people sell in this field. Particularly the ones which have chronically bled other rich parties in the game (Roche) potentially costing them billions to address through hostile take overs.

One thing I currently miss as a former emPCR user gone to clusters on MiSeq is the ability to use no amplification libraries and let emPCR capture these limited libraries and enrich them for better sequencing performance. The decoupling of emPCR from detection can be helpful for low input libraries. Currently the MiSeq requires you take 10ul of library and dilute it 200 fold before clustering and only 1/10th of this dilution makes it into the flow cell (600ul load and a flow cell likely to only hold 50ul). If anyone has protocols for low amp libraries on MiSeq...I'm all ears.

[snetmcom]

Its a good paper and I'll throw my unsolicited recommendations on there as well. I think Nick was generous with the Ion platform. The Library prep, EmPCR and CAFIE correction CPU time after the run were not baked into the time comparison. Putting Ion down as a 3 hr run should really be more like 3hr of Lib+emPCR+enrichment+quant up front, then a 3hr run, and then a few hours of CPU to get CAFIE adjusted fastqs before you have reads you can run with. Nextera + MiSeq run should be the apples to apples time comparison and this should be normalized to read length. 300 cycles of terminator data next to 100bp of homopolymer prone extentions where on average 2.3 bases per cycle are acquired is not apples to apples. In light of this, I dont see a single thing the Ion is good at (at this time point).
I'm also seeing that the Ion data only used 90% of the data while the ILMN and 454 were using closer to 99% of the data. If the Ion data was filtered more aggressively and still had worse performance its a bit unsettling.. Is this the right way to read this are did the emulsions have higher doublets on the poisson dilution than the 454 data?

Ion is certainly improving quickly but I see 454 as a prophet for their future. 7 years later the homopolymer issues is still only 5X better and not enough to displace the indel error significantly found in the supp. 1000X higher indel error rate for ion is buried in the supplement and the Ion data appears to have an additional stranded bias the 454 platform doesnt have. My advice is to not pay too much attention to the futures people sell in this field. Particularly the ones which have chronically bled other rich parties in the game (Roche) potentially costing them billions to address through hostile take overs.

One thing I currently miss as a former emPCR user gone to clusters on MiSeq is the ability to use no amplification libraries and let emPCR capture these limited libraries and enrich them for better sequencing performance. The decoupling of emPCR from detection can be helpful for low input libraries. Currently the MiSeq requires you take 10ul of library and dilute it 200 fold before clustering and only 1/10th of this dilution makes it into the flow cell (600ul load and a flow cell likely to only hold 50ul). If anyone has protocols for low amp libraries on MiSeq...I'm all ears.

i think you bring up good points, but you might be lacking PGM experience. He did indeed show ePCR time in many of the diagrams. The PGM does not have hours for CAFIE or analysis. The fastq is ready almost immediately after the run with 314 and 316. It's only hours after the run if you are doing full alignments on the larger chips. Illumina data is always filtered. I think all the platforms do this during primary now. The homopolymer accuracy has already surpassed 454. Nothing will be easier than clusters, but imaging time scales.

[Nitrogen-DNE-sulfer]

Can anyone comment on the strand specific error specific to the PGM? Has this improved with their latest data release? Can you point to data where the Homopolymer data is better than 454? Its not in this paper or evident in data I have from 454 and what Ion has recently posted. They are improving quickly and I wish the paper had that chart as I think it would have done a lot more justice to the platforms. Ion is improving at a much faster rate than 454. I reserve judgement on growth speeds on Ion vs ILMN as one party is increasing from little to more while the other is shrinking from too much to bite size so the comparison is never fair. Whats clear to me is from a practical standpoint 2 runs per ion per day and 1 from MiSeq. Pushing 3 runs a day with Ion and all of the ancillary work requires more people which offset the costs of the cheaper box but would like to hear more...

[james hadfield]

Hi Matt,
I'd agree with Phil about reliability, everyone sucks! My experience with almost every new technology has been the first few are unreliable, manufacturing catches up with R&D and the fiex gest into production machines tehn things start working after six months or so. Our first GA1 installed in 2007 was not great from data quality and yield perspectives, but it was fun.
I've had machines replaced in the first year after install, but it is hard to make a case. Record every instance of something going worng and evidence the impact in lost time or opportunity.
James.

[ramujana]

That's right, but bear in mind there were very few bases with that quality in the dataset (c.f. top panel of Figure 1). I tried to represent this with the area of the point, but I'm not sure I did a very good job getting the dynamic range across.

Recently i've got a response from LifeTechnologies representatives - what would be your comments regarding this:

Loman’s paper is very popular thanks to Illumina. However, we have some things to say about it:

1. These type of papers comparing platforms are usually not very helpful, as the peer-review process delays publication too much to consider them state-of-the-art information. It might not be so critical for established technologies (Illumina), because their potential improvements are necessarily limited. However, 6 months in a technology that is evolving so fast as Ion Torrent does is too much. The paper uses old (July 2011), short read-length (100nt) chemistry, which is completely obsolete by now.
2. The authors haven’t consider the operator bias, comparing results obtained in their lab with others generated by the manufacturers (other hands, different lab)
3. The authors haven’t considerer the analysis (filtering and trimming) bias. There is no information on how did Illumina analyze the data generated (or even, how many runs did they perform to generate them).

Life Technologies has sent a letter to Nature Biotechnology’s editor highlighting these facts (attached). I’ll let you know in case we receive any answer.

In parallel we’ve done 3 experiments:

1. We’ve repeated the run using the same sample with our current 200nt chemistry and we’ve obtained homopolymer accuracy (which has improved a 450% in the last 9 months), the average raw accuracy (99.6% in May 2012 vs 97.2% in July 2011), and the % of QV30 bases (99,9% accuracy).
2. We’ve repeated the run using a pre-launch version of our 2H 2012 chemistry showing average accuracy (99,7%) and % of QV30 bases, now with a 300nt read length.
3. We’ve used all these data for de novo genome assembly and compared it to Illumina’s results. This is what really makes sense. QV values are just the result of applying a manufacturer generated algorithm; you can give them the value you want. But how well do data work when pushed to generate meaningful results? Ask assembly metrics (number of contigs _fewer is better_ or contig length_longer is better) and you’ll have the clue.

[krobison]

It is a bit disingenuous of Life Tech to complain that Loman's paper used chemistry nobody is using right now, but then tout a chemistry that nobody can use right now.

[Chipper]

Just like what Illumina does with the
miseq...

[Joann]

Chemistry?

Look, it's a no brainer that significant clinical implementation of these platforms calls for a set chemistry protocol, not an evolving one. That's why some focus on external calibration approaches (like in-bred flies) is important even if it could allow for a competing machine, etc. to look better.