Validation of RNAseq Experiments by qPCR?

This post is in response to a couple of twitter discussions regarding whether its "useful" to do qPCR validation of RNAseq hits:

As preliminary data for a talk can I graph transcript counts of particular genes (transcriptome data) instead of qPCR (haven't done it yet)?
— Sciencegurl (@sciencegurlz0) October 27, 2014

In response to the that I posted a response I made in response to a similar query from a reviewer for a recent manuscript (original available here). Based on the fact that this came up a couple times in two weeks, I thought that I'd try to be more clear on my thoughts (and put them somewhere easier for others to find). This is how we address this issue experimentally, and in response to reviewer requests. Feel free to use any of these arguments yourself but your mileage with your supervisor/manuscript or grant reviewers may vary. Most importantly, if you have some suggestions/data/papers that we should include please comment below and I'll try to keep this post up to date.

The Answer We Gave:

We had considered performing qPCR studies to ‘re-validate’ some of our gene-expression findings but there is little evidence that qPCR analyses from the same samples will add any extra utility to our data so we decided to eschew those experiments. Previous studies have shown extremely close correlations between qPCR and RNAseq data [1-4]. Ideally, we would re-validate our findings (potentially by qPCR) in a separate cohort of samples, but due to the difficulty in accessing these samples, those experiments are not possible at this time.

What Do We Mean By Validation of RNAseq Results?

RNAseq, like microarrays before them generate a lot of data. Ideally you can determine the levels of every gene/transcript/exon in the genome, and given proper experimental design determine a large number of significantly differentially expressed genes. To follow up these findings, we often want to test how valid those observations are. In my interpretation, this can mean a few different things:

1. Are these transcripts really differentially expressed in these samples (technical reproducibility)?
2. Are these transcripts really generally differentially expressed in other samples (biological reproducibility)?
3. Do these transcriptional changes represent phenotypic differences (significance)?

For the third question, I'd say you'd want to perform some non-transcriptional response such as a western blot, enzymatic or cell based assay to show that the transcriptional change has some biological phenotype. Ideally, you may even go farther and manipulate the expression of a gene of interest back to the control condition, to test the hypothesis that a change in that gene is causative of some particular phenotype. 

Why was qPCR the traditional validation experiment from microarray studies?

Normally though, the question of validation stems from one of the first two questions. This is probably based on prior work with microarrays. Microarrays were/are great tools for transcriptomic analysis (though I am hard pressed to think of a reason to do them in lieuw of RNAseq). One major problem with microarrays was probe bias. What this means is that there was a limited number of hybridization probes in a microarray experiment, and its possible that this probe may not be representative of the transcript as a whole. Furthermore, not all transcripts of interest may be present on the microarray of choice. As a result, the standard in the field was to examine transcript abundance by qPCR to technically validate microarray results.

How similar are RNAseq and qPCR results?

Probe bias, poor sensitivity and reduced linear range are not as problematic in RNAseq experiments, since the entire transcript is assessed in a more or less unbiased manner [5]. Several studies have compared RNAseq results to qPCR data, and have found excellent correlation between these methods [1-4]. In cases where there are discrepancies exist, I would argue that it is most likely due to bias in the qPCR experiment (which has its own probe-bias based on what region of the cDNA is amplified). Therefore, it is unlikely to yield new information, and when it does, the information is probably worse than the quality of RNAseq data.

Under What Conditions Would qPCR Be a Good Validation Method?

This isn't to say qPCR isn't useful, we use it all the time in my lab. Its a great tool for looking at a small number of genes in samples for example. But when would it be good to use in the context of validating a RNAseq experiment? I would argue that it is most useful in the case where you have independent samples from those that you did your RNAseq studies on. For example, maybe for economical reasons, you examined 5 control samples and 5 drug-treated samples but you have another 20 samples available. qPCR would be a great way to test whether the differences observed are also true in separate samples, thus answering question #2. I think this is especially important as in my experience a lot of RNAseq studies are underpowered to answer the questions asked (if you want a quick and easy way to check the power of your experimental design I like Scotty).

References

Thanks to Matthew MacManes (@peroMHC) and Alejandro Montenegro (@aemonten) for spurring this discussion, and sending me towards a nice paper by Timothy Hughes [6], which is a great summary of similar, more general issues.

1. Griffith M, Griffith OL, Mwenifumbo J, Goya R, Morrissy a S, et al. (2010) Alternative expression analysis by RNA sequencing. Nat Methods 7: 843–847. doi:10.1038/nmeth.1503.
2. Asmann YW, Klee EW, Thompson EA, Perez E a, Middha S, et al. (2009) 3’ tag digital gene expression profiling of human brain and universal reference RNA using Illumina Genome Analyzer. BMC Genomics 10: 531. doi:10.1186/1471-2164-10-531.
3. Wu AR, Neff NF, Kalisky T, Dalerba P, Treutlein B, et al. (2014) Quantitative assessment of single-cell RNA-sequencing methods. Nat Methods 11: 41–46. doi:10.1038/nmeth.2694.
4. Shi Y, He M (2014) Differential gene expression identified by RNA-Seq and qPCR in two sizes of pearl oyster (Pinctada fucata). Gene 538: 313–322. doi:10.1016/j.gene.2014.01.031.
5. Wang Z, Gerstein M, Snyder M (2009) RNA-Seq: a revolutionary tool for transcriptomics. Nat Rev Genet 10: 57–63. doi: 10.1038/nrg2484.
6. Hughes TR (2009) “Validation” in genome-scale research. J Biol 8: 3. doi:10.1186/jbiol104.

What Should Be Done about Reproducibility

A recent Commentary and linked editorial in Nature regarding reproducible science (or rather the lack thereof in science) has been troubling me for a few days now. The article brings to light a huge problem in the current academic science enterprise.

What am I talking about?

In the comment, two former Amgen researchers describe some of the efforts of that company to reproduce "landmark studies" in cancer biology. Amgen had a team of about a hundred researchers called the reproducibility team and their job was to test new basic science findings prior to investing in following up these targets. Shockingly, according to the authors, only 6/53 of these landmark studies were actually reproduced. When things were not reproduced they contacted the authors to attempt to work through the potential problems. This is an incredibly dismal 11% reproducibility rate!

Could it really be that bad?

The first problem is what exactly is meant by reproducibility. In the commentary the authors acknowledge that they did attempt to use additional models in the validation process and that technical issues may have under-lied some of these differences. They also point out that their sample set is biased with respect to the findings. These were often novel and cutting edge type findings and typically more surprising than the general research finding. Also, their definition of reproducibility is unclear. If researcher says drug X has a 10 fold effect on something and the Amgen guys say it has a 3X effect on the process is that a reproducible finding. My initial reaction was that the 89% were thing where the papers said something like thing X does thing Y and there was no evidence supporting that. We don't know, and in a bit of an ironic twist, since no data is provided (either which papers were good and which were bad, or within those, which findings were good and bad) this commentary could be considered both unscientific and non-reproducible itself (also we are awfully close to April Fools Day).

So there is some bad papers out there, who cares?

Reproducibility is at the heart of everything we do as scientists. No one cares if you did something once and for reasons you cant really explain, were never able to do it again. If something is not replicable and reproducble for all intents and purposes it should be ignored. We need measures of these to be able to evaluate research claims, and we need context specificity to understand the breadth of claims. Ill toss out few reasons why this problem really matters both to those of us who do science, and to everyone else.

This is a massive waste of time and money

From the commentary:

Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis.

Wow, really? Whole fields have been built on these? In a way I don't feel bad for these fields at all. If you are going to work in a field, and are never going to bother even indirectly testing the axioms on which your field is built then you are really not so good at the science. If you are going to rely on everyone else being correct and never test it then your entire research enterprise might as well be made from tissue paper. More importantly, if you are on top of these things you are going to waste time and money figuring out not to follow this up. Hopefully this is the more common case. This really goes back to the difficulty in publishing negative data to let people know which conditions work and which don't.

The reward system for science is not in sync with the goals of the enterprise

Why are people publishing things that they know only happen one out of six times? Why are they over-extending their hypotheses and why are they reluctant to back away from their previous findings? All of these things are because we are judged for jobs and for tenure and for grants on our ability to do these things. The person who spends 3 years proving that a knockout mouse model does not actually extend lifespan walks away with nothing, the one who shows it (even if done incorrectly) gets a high impact paper and a job. Even if it didn't take an unreasonable amount time and effort to publish non-reproducible data, the risk of insulting another researcher or not contributing anything new might be enough to prevent this. Until the rewards of publishing negative or contravening data are on par with the effort, people just won't do it.

This reflects really poorly on science and scientists

Science is always and probably has always been under some type of "attack". Science as an entity and scientists as their representatives need to not shirk this off or ignore it. We have to deal with this problem head-on, whether it be at the review level or at the post-publication level. People who are distrustful of science are rightful to point at this and say, why are we giving tens of billions of dollars to the NIH when they are 89% wrong. Why not just give that money to Amgen, who seem to be the ones actually searching for the truth (not that they will share that data with anyone else).

Can anything be done?

The short answer is its really going to be difficult and its going to rely on a lot of moving parts. Reviewers should (and in my experience do) ask for explicit reprodicibility statements in the papers. This can go farther, if someone says this blot is representative of 5 experiments then there is no reason the other 4 couldnt be put in the supplement. If they looked at 100 cells and show just one, then why cant the rest be quantified in some way. Post-publication, there should be open (ie not just in lab meetings) discussion of papers and the problems and where they match or mismatch with the rest of the literature. Things like blogs and the Faculty of 1000 are great, but how often have you seen a negative F1000 review? Finally, eventually there ought to be some type of network of research findings. If I am reading a paper, and I would like to know what other results agree or disagree with this, it would be fantastic to get there in a reasonable way. This is probably the most complicated, as it requires not only publication of disagreeing findings, but also some network to link them together.



Begley, C., & Ellis, L. (2012). Drug development: Raise standards for preclinical cancer research Nature, 483 (7391), 531-533 DOI: 10.1038/483531a


What Should Be Done about Reproducibility by Dave Bridges is licensed under a Creative Commons Attribution 3.0 Unported License.