How will SAM evolve to handle multiple tags per fragment?

[krobison]

A bit of an open question looking for debate.

SAM format has a bunch of support for mate pair / paired end sequences in terms of flags and specific fields.

A number of sequencing approaches can generate multiple tags per fragment. For example, Polonator reads are really mate quads. Complete Genomics takes this even farther. Conceptually, SOLiD and perhaps Illumina could generate 4 tags from "jumping" libraries. Helicos and PacBio (and presumably the VisGen/Life technology) can use pulses of unlabeled nucleotides to potentially generate a very large number of linked reads.

Any thoughts to how to accommodate these? Or will a new format be required?

[nilshomer]

A bit of an open question looking for debate.

SAM format has a bunch of support for mate pair / paired end sequences in terms of flags and specific fields.

A number of sequencing approaches can generate multiple tags per fragment. For example, Polonator reads are really mate quads. Complete Genomics takes this even farther. Conceptually, SOLiD and perhaps Illumina could generate 4 tags from "jumping" libraries. Helicos and PacBio (and presumably the VisGen/Life technology) can use pulses of unlabeled nucleotides to potentially generate a very large number of linked reads.

Any thoughts to how to accommodate these? Or will a new format be required?

I mentioned this a while ago since we have had triple-end data for quite some time now (we don't generate such data anymore since it is not cost-effective). Anyhow, in my opinion there will have to be a non-backwards compatible SAM format, since the FLAG field among other fields are not compatible given n-linked reads. The fields that hold information about the pair/mate will have to become dynamic given the same structure. Maybe we could have a field that says how many tags are linked and then the rest of the line is easily reconstructed.

A very real problem is that rapidly iterating across the links will become difficult when the SAM file is sorted by coordinate. This can be currently seen when trying to iterate across multiple alignments (some on different chromosomes).

It would be helpful if the sequencing companies could help in this discussion by giving us insight into what they intend to release so that we can remain proactive in developing good open formats.

My 2.097 cents CAD.

[krobison]

How did you generate "triple-end" data? Do you have access to the magic, unreleased SOLiD reagents for reading "backwards" ?

[lh3]

We used to discuss this issue. A potential solution is to keep all ends in a loop. e.g. in the mate position field end1 points end2, end2 to end3, and end3 back to end1. We can also keep the positions of all ends in a tag for each alignment. However, the real question is how we are going to use multi-end information. That is why it is not decided yet. But I do not see this as a major problem.

[nilshomer]

How did you generate "triple-end" data? Do you have access to the magic, unreleased SOLiD reagents for reading "backwards" ?

Illumina

Anyhow, not that exciting just saying we will have to solve this problem.

[krobison]

Heng:

This sort of data may be particularly valuable on the PacBio instrument, depending on how long a fragment you can do it on.

For example, imagine a series of ~50 nt reads separated by ~500nt gaps across a 20Kb fragment (I've totally pulled those numbers out of thin air, but I think this is roughly plausible from what I've heard). That would clearly be useful for de novo assembly, but would also help (I think) resolve complex repeat structures in resequencing applications.