Ion Torrent complains to Nature Biotech about bias in Loman paper

[ECO]

This is a couple days old but figured it would be interesting to some of you.

Nick Loman's original paper comparing desktop sequencers (which we've discussed here before)...has been subject of controversy. LifeTech apparently didn't like how the study was presented, and wrote a letter to Nature about the study.

Other takes on it:

• GenomeWeb article
• flxlex's blog

Fun stuff.

[Heisman]

And they still are not showing any data for homopolymers longer than 5bp. Makes no sense to me (besides the obvious explanation that the data looks bad).

As for their two complaints: both of them are in regards to the comparison and not the actual analysis of the Ion data. When they came here a couple weeks ago I specifically requested they generate a full error profile each time they update their reagents and make that publicly available. That includes homopolymer errors at >5bp. When they have good data, they should publish that fact, and then they won't have worry about any previous papers.

[adaptivegenome]

So no one has examined homopolymer error in ion torrent data?

[Heisman]

So no one has examined homopolymer error in ion torrent data?

I'm sure someone has but I've yet to run a sample so I haven't seen any data myself. When they came to talk to us it was just after the Nature BioTech paper came out and they were "prepared" for us to ask about it by showing us that their new reagents gave better results (similarly to what they show in the links above); but even then they only showed data up to 5bp homopolymer runs and I asked them about longer ones but they claim they were just the sales people and not "bioinformatics" so they weren't sure. Just left a sour taste in my mouth.

[adaptivegenome]

I'm sure someone has but I've yet to run a sample so I haven't seen any data myself. When they came to talk to us it was just after the Nature BioTech paper came out and they were "prepared" for us to ask about it by showing us that their new reagents gave better results (similarly to what they show in the links above); but even then they only showed data up to 5bp homopolymer runs and I asked them about longer ones but they claim they were just the sales people and not "bioinformatics" so they weren't sure. Just left a sour taste in my mouth.

I am in the same situation as you. I want to know more about the technology but it seems hard to get any information. We have inbred lines of flies that we sequenced with Illumina and we can estimate error by looking at discordance of reads at a given locus (since there should only be a single allele). I almost feel like sending a fly for sequencing by Ion Torrent so that I could just measure rates of error for sequences like homopolymers.

Update us if you learn anything.

[Joann]

We have inbred lines of flies that we sequenced with Illumina and we can estimate error by looking at discordance of reads at a given locus (since there should only be a single allele).

Wow, that's a great idea for standards assessment.

[james hadfield]

I've worked on a fair number of comparisons and they are always out of date if they get published. The paper by Nick did not suggest users buy a MiSeq over a PGM. It seemed pretty clear in the paper that the technologies are still evolving and even descrbies the benefits of the different PGM chip types for users as an advantage over other platforms.

Lets not forget that Nick's lab won their Ion PGM in a competition judged by Ion. And they don't have a Miseq (it was only just released at the time and no-one could get hold of one). I am not sure if Nick contacted Ion to run his samples on the latest internal chemistry. But the MiSeq data presented are pretty close to what we see on our MiSeq instrument.

The letter was a surprise and perhaps Alan Williams hit send immediatley after writing it rather than sleeping on it first. I think it was a mistake. But Life Tech had an awful lot of catch up to do before Ion came along. Emotions are still running high, in a game where one company could dominate in clinical sequencing.

[adaptivegenome]

I've worked on a fair number of comparisons and they are always out of date if they get published. The paper by Nick did not suggest users buy a MiSeq over a PGM. It seemed pretty clear in the paper that the technologies are still evolving and even descrbies the benefits of the different PGM chip types for users as an advantage over other platforms.

Lets not forget that Nick's lab won their Ion PGM in a competition judged by Ion. And they don't have a Miseq (it was only just released at the time and no-one could get hold of one). I am not sure if Nick contacted Ion to run his samples on the latest internal chemistry. But the MiSeq data presented are pretty close to what we see on our MiSeq instrument.

The letter was a surprise and perhaps Alan Williams hit send immediatley after writing it rather than sleeping on it first. I think it was a mistake. But Life Tech had an awful lot of catch up to do before Ion came along. Emotions are still running high, in a game where one company could dominate in clinical sequencing.

I agree. I hope more people contribute comparisons particularly of the error profiles.

[KevinLam]

I never thought I would live to see 'sour grapes' as a tag in a post here.

To be fair, I do not think platform bias should surprise anyone. Perhaps the knee jerk reaction is that PGM isn't good for xxx experiment as it has xxx bias is unfair. One statement that I thought was pretty funny was 'PGM has no issues with homopolymer at >100x coverage'

the person who said it mentioned softly that there are other issues at play which I have been trying to find out more BUT I must say that knowing the characteristics of the platform isn't necessarily a bad thing. if it is a consistent systemic error profile, then it can be corrected for.
It's the out of the ordinary things that can't be corrected for.

I think sufficient human whole genome sequencing has been done to know that the best way to correct for platform error on one system .... is to sequence with ALL platforms.

[IonTorrent]

Just to add to the discussion here from an Ion Torrent perspective...

Data Access - There are representative data sets that are all date stamped and available for download at Ion Community in the Torrent Dev section. These are available for scientific inspection or can be used as input for software development. Please go leverage this resource. Make as any comparisons and measurements about performance as you'd like. There are a lot of ways to look at performance data. That's why we make the data available.

There are three new runs there now. One from our pending-release 200-250 bp chemistry (BEA-638) and another from long read chemistry (300-400 bp) coming later this year (B7-143). And lastly, we just put up a 2 GB Paired End Run from a 318 chip for the latest round of the Grand Challenge. The data are available, please dig in.

Regarding the Pallen publication, it's a scientific publication. We thought the controls around data acquisition and data filtering could have been slightly tighter. This is just a simple observation that we think impacts the statements about accuracy within the publication. Of course, the argument about old kits is easily addressed with the current performance data released above. Everyone can see what we are doing today and tomorrow with this recent data release.

The technology works. I encourage everyone to dig into the data to see for themselves. Please explore the quality and accuracy of our fast sequencing & long read chemistry.