Samtools Flag to identify outward facing (mate-pair not paired end) reads

[lesander]

Hi,
Can someone tell me what flag I should grep for in order to
obtain reads that are outward facing (mate pairs: <--- && ---> ) and what
flags I should grep for in order to obtain reads that are inward facing (paired ends: --> && <---)?
many thanks
Les
p.s. I have consulted the picard's useful site at http://picard.sourceforge.net/explain-flags.html
but what confuses is me that the definition of "proper pair" etc which is dependent on the aligner's definition based on the insert size.

[maubp]

Right now, I don't think this is possible.

The FLAG doesn't record the pairing direction, just if the aligner/assembler says it was correct or not.

In principle this could be recorded as part of the read group @RG line, but I don't think that has been standardised yet. You could raise this on the samtools-devel mailing list.

[lesander]

Right now, I don't think this is possible.

The FLAG doesn't record the pairing direction, just if the aligner/assembler says it was correct or not.

In principle this could be recorded as part of the read group @RG line, but I don't think that has been standardised yet. You could raise this on the samtools-devel mailing list.

Can someone please confirm if the following is correct:

I thought, 99,147 and 83,163 are the flags for "proper pair" (i.e. --> <--)
within insert size.

For mate-pair (outward facing) the flag is 115, 179 for concordant (within insert size)
and 113,177 for discordant.

thanks

[maubp]

The problem is "proper pair" is context dependent, it could be --> <-- for Illumina paired end, <-- --> for Illumina mate pairs, or --> --> for Roche pairs, or --> --> ---> ... --> for strobed reads.

What you can do is look at the FLAG as see for any pair of reads were they --> <-- etc, and if they were "properly paired", but that just tells you how they mapped, not what the library construction should have been. This also won't help for when only one of a pair is mapped, or neither.

Perhaps I misunderstood you - that may be all you want to do?

[swbarnes2]

Can someone please confirm if the following is correct:

I thought, 99,147 and 83,163 are the flags for "proper pair" (i.e. --> <--)
within insert size.

For mate-pair (outward facing) the flag is 115, 179 for concordant (within insert size)
and 113,177 for discordant.

thanks

That second set of numbers doesn't correspond to how I thought mate pairs work. I thought that mate pairs were always outward-running. So the numbers would be 97,145,81, and 161, if your aligner is expacting ordinary paired-end reads. If you rev-comped your reads, they'd be inward running, and would look like ordinary paired end reads.

[kmcarr]

Can someone please confirm if the following is correct:

I thought, 99,147 and 83,163 are the flags for "proper pair" (i.e. --> <--)
within insert size.

For mate-pair (outward facing) the flag is 115, 179 for concordant (within insert size)
and 113,177 for discordant.

thanks

No, 115 & 179 (or 113 & 177) are nonsensical if you are expecting outward facing mate-pairs. Those flag values would have both read 1 and read 2 aligned to the minus strand. A proper mate-pair would have the reads on opposite strands.

A proper set of mate-pair reads would have exactly the same flag values as a proper set of paired-end reads (83,163 or 99,147). It is up to the alignment software to decide whether the pair is proper based on relative orientation and distance of reads 1 & 2 and the stated methodology.