Spike in dataset to compare DESeq vs cuffdiff

narges · 09-06-2013, 05:15 AM

Hi all,

I wanted to ask if you have any idea to be fair as much as possible when comparing two RNA-seq analysis methods like DESeq and cuffdiff. I know one ideal option is using spikein datasets, or alternatively a simulated dataset but then I have no clue about if there are any publicly available spikein dataset for RNA-seq.
Any help or idea would be appreciated.

Wolfgang Huber · 09-07-2013, 04:47 PM

Dear Narges

simulation is great to check how well your software does (or approximates) what your theory says, but it does not help with the question whether your theory (or: model) agrees with reality.

As for real data, two criteria that seem to make sense are specificity (how many or few false positives do you find) and sensitivity (how many true positives do you find). As for specificity, this is in fact quite easy with real data, just do the same comparison that you would like to do, but in a "mock" fashion with all samples actually being biological replicates. For sensitivity, you need a "ground truth" of truly differentially expressed genes. These may be hard to come by - but you can use prior biological knowledge, independent experiments, etc. Also, this ground truth does not need to be strictly true for the purpose of method ranking, it is sufficient if it is enriched for truth, and the false genes in there are random (the concept of 'pseudo-ROC' by Richard Bourgon).

Best wishes
Wolfgang

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Can I use cuffdiff to compare 2 groups?	twotwo	Bioinformatics	2	06-24-2013 06:52 AM
Different RPKM values in same dataset using Cufflinks or Cuffdiff (v1.3.0)	hlwright	Bioinformatics	3	11-09-2012 04:51 AM
sample RNAseq data to compare cuffdiff and DESeq performance	Sun-SEQ	Bioinformatics	0	08-15-2012 10:13 PM
DESeq without biological replicates (dataset: Marioni et al.)	Andrea Apolloni	Bioinformatics	3	01-13-2012 05:58 AM
compare expression? cuffcompare or cuffdiff	vebaev	RNA Sequencing	6	12-21-2010 10:59 PM

09-06-2013, 05:15 AM	#1
narges Member Location: Finland Join Date: Aug 2012 Posts: 29	Spike in dataset to compare DESeq vs cuffdiff Hi all, I wanted to ask if you have any idea to be fair as much as possible when comparing two RNA-seq analysis methods like DESeq and cuffdiff. I know one ideal option is using spikein datasets, or alternatively a simulated dataset but then I have no clue about if there are any publicly available spikein dataset for RNA-seq. Any help or idea would be appreciated.

09-07-2013, 04:47 PM	#2
Wolfgang Huber Senior Member Location: Heidelberg, Germany Join Date: Aug 2009 Posts: 109	Dear Narges simulation is great to check how well your software does (or approximates) what your theory says, but it does not help with the question whether your theory (or: model) agrees with reality. As for real data, two criteria that seem to make sense are specificity (how many or few false positives do you find) and sensitivity (how many true positives do you find). As for specificity, this is in fact quite easy with real data, just do the same comparison that you would like to do, but in a "mock" fashion with all samples actually being biological replicates. For sensitivity, you need a "ground truth" of truly differentially expressed genes. These may be hard to come by - but you can use prior biological knowledge, independent experiments, etc. Also, this ground truth does not need to be strictly true for the purpose of method ranking, it is sufficient if it is enriched for truth, and the false genes in there are random (the concept of 'pseudo-ROC' by Richard Bourgon). Best wishes Wolfgang __________________ Wolfgang Huber EMBL