RNA seq data filter low count before or after normalization

[qliu]

Hi there,

I just created login, my first time post here : )

I am trying to figure out which step should I filter the gene with low count.
before normalization, after normalization? and how to determine the optimal cutoff?

If you know any literature and document that address my question, please let me know.

thanks a lot.

Q

[dpryan]

Normally you do it after normalization, though there's typically not much difference to doing it before vs. after. Regarding the cut off, please see the genefilter package and the accompanying paper in PNAS.

[dariober]

Originally Posted by qliu

Hi there,

I just created login, my first time post here : )

Hello!

If you know any literature and document that address my question, please let me know.

(Assuming you want to do differential gene expression expression analysis and you have RNAseq or similar type of data)

The edegR vignettes is a document where filtering is applied. I don't know of systemic studies addressing where to set the cutoff.

I am trying to figure out which step should I filter the gene with low count.
before normalization, after normalization? and how to determine the optimal cutoff?

The vignettes quoted above apply the filter after transforming raw counts to counts per million.
In practice I think most of us apply a common sense threshold. If say a gene has 1 cpm (counts per million) in all the libraries, that gene can't be biologically interesting. The reason to reject genes early, i.e. before testing for de, is to make the adjustment for multiple testing less "aggressive".

[dpryan]

Originally Posted by dariober

I don't know of systemic studies addressing where to set the cutoff.

See the Bourgon et al. 2010 paper in PNAS. That described microarrays, but the same applies to RNAseq. Detection power is optimized by performing the filtering on the p-values after testing. I do occasionally prefilter, but only with limma/voom when I get weird fits (sometimes that happens and you have to remove low expressors to get appropriate results).

BTW, this is why I use DESeq2, it does all of this for me.

[dariober]

Originally Posted by dpryan

See the Bourgon et al. 2010 paper in PNAS. That described microarrays, but the same applies to RNAseq. Detection power is optimized by performing the filtering on the p-values after testing. I do occasionally prefilter, but only with limma/voom when I get weird fits (sometimes that happens and you have to remove low expressors to get appropriate results).

BTW, this is why I use DESeq2, it does all of this for me.

Thanks - I'll have a look!