SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
DESeq problems janec Bioinformatics 17 08-05-2013 04:44 AM
pval/padj equals 0 using DESeq jorgebm Bioinformatics 4 05-10-2012 12:19 AM
edgeR - select DE based on pval and FC crh Bioinformatics 0 08-02-2011 08:20 AM
Is more than two conditions possible in DESEQ? greener RNA Sequencing 5 05-09-2011 03:10 PM
interpreting DESeq result, padj - values Azazel Bioinformatics 0 10-06-2010 05:49 PM

Reply
 
Thread Tools
Old 09-04-2012, 06:00 AM   #1
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Post DESeq: pval vs padj

Hello everybody,

Statistics is not my strong side, so I'm asking a basic question here. I've done some bacterial RNA seq. I was able to extract my reads with HTSeq and I've done some statistics with DESeq. Now I have a list of DE genes with their associated p value and padj value. I know from the DESeq vignette that padj value corresponds to p-value adjusted for multiple testing using Benjamini-Hochberg method. However, for me that's equivalent to Chinese, i.e. I'm not sure what does it actually means.

Which p value I should consider when treating my data? Number of genes I would include in a qPCR validation study will change notably... And I would also like to understand why I should consider one or the other parameter.

Thanks in advance!
ThePresident is offline   Reply With Quote
Old 09-04-2012, 06:52 AM   #2
dpryan
Devon Ryan
 
Location: Freiburg, Germany

Join Date: Jul 2011
Posts: 3,480
Default

You'll want to use the adjusted p-value. For the reason why, I would suggest that you review what a p-value is and why you expect to find spurious findings with increasing numbers of tests (the various adjusts are aimed at addressing this).
dpryan is offline   Reply With Quote
Old 09-04-2012, 07:00 AM   #3
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Default

Many thanks. I'll review the subject in some statistics handbook for details.
ThePresident is offline   Reply With Quote
Old 09-04-2012, 09:33 AM   #4
mgogol
Senior Member
 
Location: Kansas City

Join Date: Mar 2008
Posts: 197
Default

I always thought this was a pretty good illustration:

http://xkcd.com/882/
mgogol is offline   Reply With Quote
Old 09-05-2012, 06:49 AM   #5
ishmael
Member
 
Location: NY, US

Join Date: Jul 2008
Posts: 17
Default

http://www.nature.com/nbt/journal/v2...1209-1135.html
I think this is a good start point.

PS, I am Chinese, but I don't think statistics is so simple as Chinese :-)
ishmael is offline   Reply With Quote
Old 09-05-2012, 07:58 AM   #6
mbblack
Senior Member
 
Location: Research Triangle Park, NC

Join Date: Aug 2009
Posts: 245
Default

Quote:
Originally Posted by ThePresident View Post
Many thanks. I'll review the subject in some statistics handbook for details.
just do a web search for some simple terms like "p-value versus FDR" and you will find many good summaries, many of them on various stats departments or professors web pages.

A fundamental difference is that a p-value is a statement about the probability of an observed test statistic given its distribution.

While an FDR is a statement about the probability of false discoveries given a certain number of simultaneous tests and their p-value distribution. It is an attempt to control for false discoveries as the type I error tends to balloon with multiple tests, and that multiplicity of errors is not reflected in the individual test statistic's p-values.

So if you base your selection on p-values, you will end up inherently including a large number of false positives. Using the FDR, you are controlling the number of false positives across all your significant statistical tests.
__________________
Michael Black, Ph.D.
ScitoVation LLC. RTP, N.C.
mbblack is offline   Reply With Quote
Old 09-05-2012, 10:47 AM   #7
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Default

Quote:
Originally Posted by mbblack View Post

A fundamental difference is that a p-value is a statement about the probability of an observed test statistic given its distribution.

While an FDR is a statement about the probability of false discoveries given a certain number of simultaneous tests and their p-value distribution. It is an attempt to control for false discoveries as the type I error tends to balloon with multiple tests, and that multiplicity of errors is not reflected in the individual test statistic's p-values.

So if you base your selection on p-values, you will end up inherently including a large number of false positives. Using the FDR, you are controlling the number of false positives across all your significant statistical tests.
Well explained, thanks! In fact, that's what I did; now I do have somewhat better understanding of what those parameters are. Only one thing: how do you interpret padj value in terms of significance? You consider that a null hypothesis is rejected under some threshold (like for p value) or...? I don't know if I was enough clear...
ThePresident is offline   Reply With Quote
Old 09-05-2012, 11:14 AM   #8
mbblack
Senior Member
 
Location: Research Triangle Park, NC

Join Date: Aug 2009
Posts: 245
Default

You'll tend to interprete the FDR value similarly to how you interprete a single p-value. What is your comfort level in terms of false positives? Also, what are you generating gene lists for?

So, for example:

scenario one - you wish to pull out significantly differentially expressed genes for some sort of ontology or enrichment analysis. Your primary goal is to identify biological processes, pathways or other ontology categories. So you may be fairly relaxed with your choice of cutoff in order to be sure to have sufficient genes to get a reasonably robust enrichment result. So, you may pick an FDR of < 0.05, or even 0.1 if you need to pad out your gene lists.

scenario two - you are trying to pick out genes as candidates for bio-assay development, so you'd like to find the least number necessary to characterize your system, and you need to be stringent about your risk of false positives (wasted money down the road if those fail to validate for your assay). So you now pick a more stringent FDR, maybe even going to < 0.01 if that gives you enough to continue with. Perhaps you simultaneously throw in a fold change cutoff as well, so only take genes with both an FDR < 0.05 and a log2 FC > 2 (picking only highly significant high expressors).

So, as with any choice of statistical criteria, you pick a cutoff that makes sense in light of your questions(s) and your system.

Since I work in toxicology, we tend to worry more about false negatives than false positives, so I generally need not be overly stringent with my cutoff and usually use an FDR < 0.05 to 0.1 in order to be sure I capture enough genes for my downstream analyses (and I frequently add a fold change filter as well, with linear fold change of +/-1.5 to +/-2) - but it really depends on what you want out of the diff. gene expression analysis in the first place.
__________________
Michael Black, Ph.D.
ScitoVation LLC. RTP, N.C.
mbblack is offline   Reply With Quote
Old 09-05-2012, 11:31 AM   #9
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Default

Honestly, I don't have any certitudes concerning the outcome of my experience. I have my control and my test condition. Some genes could be upregulated, others downregulated but I expect that a majority would be unaltered.

From there, and considering your examples, I should expect some of the genes to come necessarily as false positives. So, a prudent way to proceed would be to consider a more stringent approach... using FDR < 0.05 I got 15 genes which is fair well plus I'm confident that I see some true changes in expression.

Thank you for your help
ThePresident is offline   Reply With Quote
Old 09-05-2012, 11:57 AM   #10
mbblack
Senior Member
 
Location: Research Triangle Park, NC

Join Date: Aug 2009
Posts: 245
Default

Quote:
Originally Posted by ThePresident View Post
Honestly, I don't have any certitudes concerning the outcome of my experience. I have my control and my test condition. Some genes could be upregulated, others downregulated but I expect that a majority would be unaltered.

From there, and considering your examples, I should expect some of the genes to come necessarily as false positives. So, a prudent way to proceed would be to consider a more stringent approach... using FDR < 0.05 I got 15 genes which is fair well plus I'm confident that I see some true changes in expression.

Thank you for your help
In any genomic differential gene expression experiment, one expects the considerable majority of genes will not be differentially expressed. That's just biology.

The reality also is you will always have some false positives and some false negatives in any large scale statistical analysis since you cannot either truly eliminate all errors (type I and type II), nor could you really know if you had should you try to do so. Using tools like FDR helps to control those types of errors, but it does not eliminate them.

Also, replicates come into play too. With too few replicates, your statistical tests have limited power, so you have few statistically significant test results. That in turn means your FDR corrected p-values also will have very few (often none, if you had no replicates or only one or two) significant results, because your p-value distribution did not reflect any discrimination.

If you actually have no replicates, then it really is pointless to even bother computing the statistics. In that worst case scenario, you'd do best by simply ranking genes by normalized expression or raw counts, and pick those with the greatest difference in observed values (and then validate them independently).

So you have to interpret your results in light of your experimental limitations, as well as what your goal from the analysis was, and adjust things as the situation calls for. The stats are just tools to guide you and add some rigor to your analysis.
__________________
Michael Black, Ph.D.
ScitoVation LLC. RTP, N.C.
mbblack is offline   Reply With Quote
Old 09-05-2012, 12:19 PM   #11
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Default

Quote:
Originally Posted by mbblack View Post

Also, replicates come into play too. With too few replicates, your statistical tests have limited power, so you have few statistically significant test results. That in turn means your FDR corrected p-values also will have very few (often none, if you had no replicates or only one or two) significant results, because your p-value distribution did not reflect any discrimination.
Fair well. Actually, I have two biological replicates for each of my conditions. It doesn't give me a huge confidence for my stats, but I assume it's enough to call for the most DE genes. I'll probably miss some of them by being too stringent but my goal is to get those that could have some biological impact. Even if I have a gene whose expression is statistically different from my reference condition, I would not consider it unless the fold change is worth something.

If I could I would have done 3 or more replicates for each of my conditions. But the budget was limiting so... you know the story. Now, I have to work with what I have and I'm trying to use some statistical tools to help me get through and avoid a huge amount of false positives that would cost even more in downstream validation.

But I understand that statistics can me misused and that we should always consider it as a tool in light of our experiment. We often use it straightforward, losing the big picture, and falling down for a p value < 0.05 because that is enough for most peer-review journals right? Not many are going to ask for a detailed statistical analysis, so in many labs (including mine) the main goal is to get a star* (statistical significance) above your histogram.

Anyway, my English is not at the top but I hope you understood what I wanted to say.

Thanks again
ThePresident is offline   Reply With Quote
Old 09-22-2012, 12:21 PM   #12
billstevens
Senior Member
 
Location: Baltimore

Join Date: Mar 2012
Posts: 120
Default

About qPCR validation, my DESeq run calls many genes that have a differential expression difference as low as 20%, significantly differentially expressed. While I appreciate the power of DESeq, I need to verify this with qPCR.

I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?
billstevens is offline   Reply With Quote
Old 09-22-2012, 07:07 PM   #13
ETHANol
Senior Member
 
Location: Western Australia

Join Date: Feb 2010
Posts: 308
Default

In regards to qPCR validation, if you are using the same RNA as you did your RNA-seq with, it is meaningless. Well, not meaningless, it means you have controlled for technical noise but not biological noise.

pval vs. padj: This is the perspective from a biologist with very little statistical understand, but thought I might be able to add something. Fools all speak the same language. If you have 10,000 genes and you do pval cutoff of 0.2, while each one of those genes has an 20% chance of being a false positive, you will also get on average 2,000 false positives in your data set. So for example, if you got 2,200 differentially expressed genes, on average only 200 of then would be real.

Where as a padj(FDR) cutoff of 0.2 means on average 20% of the genes in you list are false positives.

If this isn't quite correct someone with a better understanding of statistics please chime in.
__________________
--------------
Ethan
ETHANol is offline   Reply With Quote
Old 09-23-2012, 03:08 AM   #14
dpryan
Devon Ryan
 
Location: Freiburg, Germany

Join Date: Jul 2011
Posts: 3,480
Default

Quote:
Originally Posted by billstevens View Post
I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?
My experience seems to parallel those of your lab mates. While it's certainly not impossible to detect <2x changes in qPCR, it's not trivial (you'll probably just need more samples). I would also question whether 20% changes in RNA are biologically meaningful. Given the number of samples that would have to be looked at to significantly discern that sort of RNA level change, I probably wouldn't bother unless the candidates were highly explanatory.
dpryan is offline   Reply With Quote
Old 09-23-2012, 01:37 PM   #15
ThePresident
Member
 
Location: Sherbrooke / Canada

Join Date: Jun 2012
Posts: 72
Default

I was just about to ask for qPCR validation. What is the routine for RNA-seq , i.e. is it necessary to validate data by qPCR? If I'm about to publish those RNA-seq results, I'm afraid reviewers will ask for qPCR validation even for significantly (<0.01) DE genes (fold change > 2). And, as I see it, it is better to use new RNA samples in order to control for biological differences...?

Thanks in advance,

TP
ThePresident is offline   Reply With Quote
Old 09-23-2012, 03:15 PM   #16
billstevens
Senior Member
 
Location: Baltimore

Join Date: Mar 2012
Posts: 120
Default

dpryan, that's good advice. I want to input my results into SPIA, and a couple of pretty cool pathways pop up, but some of thse are fold changes of 1.3 or so (very highly expressed, which is why DESeq called it). I doubt I could get it to pop up on qPCR. Would you say its fair to then say that RNA-Seq has better resolution than qPCR?

ThePresident, if you have fold changes over 2, you should do qPCR. You should be able to show fold changes and it strengthens your case. What I would say is that you do qPCR on your RNA samples, and assuming you have replicates, do qPCR on the replicates as well. Have it in triplicate, that way you are now doing technical noise (RNA-Seq vs. qPCR) and you are doing biological noise (from doing 3 replicates).
billstevens is offline   Reply With Quote
Old 09-23-2012, 03:25 PM   #17
Wallysb01
Senior Member
 
Location: San Francisco, CA

Join Date: Feb 2011
Posts: 286
Default

Quote:
Originally Posted by billstevens View Post
I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?
This depends on your experimental design. If you're just using two samples with technical triplicates, yes, >2x is about all you can hope to detect. The noise in qPCR is just that high. And if you step back and think about it, it is clear why. PCR relies on a doubling of the DNA with every round of replication, so its hard to pick up on differences less than 2x, even with triplicates. If you start using several biological replicates and expand to 5 technical triplicates you'll start see some real power.
Wallysb01 is offline   Reply With Quote
Old 09-23-2012, 05:53 PM   #18
billstevens
Senior Member
 
Location: Baltimore

Join Date: Mar 2012
Posts: 120
Default

Thanks Wallysb01, that's what I figured. But here's the question, what is "real power"? I have 4-5 biological replicates, and I will be doing it in triplicate (5 technical triplicates, and I just wouldn't have any room left on my plate). Could I get down to 1.5X? What, quantitavely, is your experience?
billstevens is offline   Reply With Quote
Old 09-24-2012, 04:07 AM   #19
mbblack
Senior Member
 
Location: Research Triangle Park, NC

Join Date: Aug 2009
Posts: 245
Default

Quote:
Originally Posted by ThePresident View Post
I was just about to ask for qPCR validation. What is the routine for RNA-seq , i.e. is it necessary to validate data by qPCR? If I'm about to publish those RNA-seq results, I'm afraid reviewers will ask for qPCR validation even for significantly (<0.01) DE genes (fold change > 2). And, as I see it, it is better to use new RNA samples in order to control for biological differences...?

Thanks in advance,

TP
Whether or not an editor or reviewer will insist on diff. gene validation is entirely dependent on just what the publications is about. If this is a some sort of general genomic characterization experiment (eg. a chemical or other exposure experiment) then very likely no. Just like such an experiment conducted using gene microarrays, as long as the experiment was properly controlled and included sufficient biological replication for robust results, nobody will be expecting validation of quite likely hundreds of putative DE genes.

However, if you are trying to characterize something more specific - a particular metabolic pathway, or you are trying to pick genes to base a bio-assay or clinical assay upon, then validation will likely be deemed a necessary step, at least for your final candidate gene selections.

But it is impossible to say whether your study really needs DE validation without knowing the specifics of what question(s) you are specifically addressing in your publication. Once you've clarified what the publication is actually about, usually the answer to validate or not is pretty much self-evident.
__________________
Michael Black, Ph.D.
ScitoVation LLC. RTP, N.C.
mbblack is offline   Reply With Quote
Old 09-24-2012, 06:28 AM   #20
Wallysb01
Senior Member
 
Location: San Francisco, CA

Join Date: Feb 2011
Posts: 286
Default

Quote:
Originally Posted by billstevens View Post
Thanks Wallysb01, that's what I figured. But here's the question, what is "real power"? I have 4-5 biological replicates, and I will be doing it in triplicate (5 technical triplicates, and I just wouldn't have any room left on my plate). Could I get down to 1.5X? What, quantitavely, is your experience?
Well, its hard to say exactly how much power you'll have without knowing what kind of biological and technical noise you're dealing with in your system. But in my experience ~15 biological x technical replicates and you have a good chance at 1.5x becoming statistically significant. It will help if your genes are fairly highly expressed and easy to amplify. When this many replicates fail for me, its often because the ct-values are up close to 30 or more and jump around too much. When ct-values are <25 the technical noise is usually quite low, at least in my experience.

How many genes are you trying to do this with? I highly doubt anyone is going to expect this kind of validation across an entire RNA-seq experiment. Is the issue that a few of you genes of highest interest do not show much fold change?
Wallysb01 is offline   Reply With Quote
Reply

Tags
deseq, pvalue

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 08:38 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO