What does biologically meaningful mean? A perspective on gene regulatory network validation
Gene regulatory networks (GRNs) are rapidly being delineated, but their quality and biological meaning are often questioned. Here, I argue that biological meaning is challenging to define and discuss reasons why GRN validation should be interpreted cautiously.
KeywordsGene Regulatory Network Green Fluorescent Protein Expression Transgenic Green Fluorescent Protein Clone Resource Conformation Capture Technique
The control of gene expression is pivotal in biology. It is accomplished by a large number of regulators, including transcription factors (TFs), that can modulate mRNA synthesis by directly interacting with regulatory genome sequences. The human genome contains about 20,000 genes and an estimated 1,400 TFs . Although much is known about the basic mechanics of transcription, little is known about how TFs function collectively in the context of intricate gene regulatory networks (GRNs) to achieve complex biological outputs during development and in physiology and disease.
Ideally, one would like to comprehensively map the binding of each TF within the genome and understand the effects that such interactions have on its target genes. Conversely, for each gene, one would like to know which TFs contribute to its expression and in which cells and under which circumstances this contribution occurs.
Overview of commonly used techniques for gene regulatory network mapping and their advantages and limitations
Computational inference (reverse engineering)
Infers putative regulatory relationships from gene expression data
The interactions predicted could be indirect (they do not have to reflect physical interactions); regulators that themselves do not change in expression will be missed; detection limits of mRNA measurements will affect GRN predictions (regulators or genes expressed at low levels will be missed)
Chromatin immunoprecipitation (ChIP)
Experimentally identifies physical interactions between TFs and DNA; TF-centered (protein-to-DNA)
In vivo; can detect TF dimers and complexes
Condition-dependent interactions can be missed; needs high-quality, highly specific ChIP-grade antibodies; when a universal tag is used for immunoprecipitation, TF is usually overexpressed; peak calling required
Yeast one-hybrid (Y1H)
Experimentally identifies physical interactions between DNA and TFs; gene-centered (DNA-to-protein)
Heterologous; condition- and context-independent
TFs that require post-translational modifications before binding DNA will be missed; not yet suitable for heterodimers
Gene regulatory networks
GRNs are graph diagrams that depict interactions between genes and their regulators (such as TFs). These interactions can indicate a regulatory relationship, and/or can depict a physical interaction between a TF and a genomic DNA region associated with a particular gene. A genome-scale method used for inferring regulatory interactions is to computationally search transcriptomic data for correlations between gene and TF expression. This reverse engineering approach has been pioneered in yeast  and has also been applied to mammals [9, 10]. ChIP starts with a protein and is an example of a TF-centered (protein-to-DNA) method, whereas Y1H starts with a DNA fragment and can be referred to as a gene-centered (DNA-to-protein) technique (reviewed in [6, 7]). Both ChIP (for example, [11, 12, 13, 14, 15, 16]) and Y1H assays (for example, [17, 18, 19, 20, 21]) have been successfully used in various systems and have each led to a wealth of data. Importantly, inferred regulatory relationships are not necessarily a result of direct physical interactions. Conversely, for physical interactions between TFs and DNA, the regulatory consequence (repression or activation) is usually not known. Therefore, for optimal coverage and information content, both types of approaches need to be applied and integrated.
What is 'biologically meaningful'?
The quality of GRNs depends on the proportion of real interactions that are retrieved and the proportion of retrieved interactions that are real. For many scientists, interactions identified by high-throughput methods are deemed biologically meaningful (real) only if a regulatory and/or functional consequence is demonstrated, after which the interaction is considered 'validated in vivo'. However, a lack of in vivo validation does not necessarily invalidate an interaction because: (i) in vivo validation methods have their own limitations; (ii) a TF binding event might have been attributed to the wrong gene - for instance, when a TF binds an enhancer far from a transcription start site; and (iii) biological safety nets that buffer the loss of individual TFs can mask the effect of genuine DNA-TF interactions on gene expression.
Data quality: false negatives
Data quality: false positives
False positives can be incorporated into GRNs and can be either technical or biological. Technical false positives are interactions that are sporadic in nature and cannot be repeated, even with the same assay with which they were originally retrieved. Obviously, any high-throughput method should avoid detecting spurious interactions by carefully optimizing and evaluating the robustness of the assay. Biological false positives are defined as interactions that are robustly detected but that are not biologically meaningful. In computationally derived GRNs spurious edges (regulatory interactions) can arise if both a TF and its inferred target are regulated by another TF that itself does not change in expression. In ChIP experiments, biological false positives might be obtained when the antibody is not exclusively specific for the TF that is being studied, or when a TF is overexpressed (for instance, from a transgene), and starts binding to lower affinity or non-specific sites. Furthermore, selecting a threshold that is too low when 'calling' interactions from a background of non-interacting fragments may result in the inclusion of false interactions. Y1H assays can retrieve interactions that do not occur in vivo when a TF binding site is available in the context of yeast chromatin but not in the organism from which the DNA fragment was cloned. TF levels in yeast are controlled by a yeast promoter and by the copy number of the TF-expressing plasmid, and it is possible that lower affinity DNA sequences are bound when TF levels are high.
Five levels of validation
There are five conceptual levels of validation of interactions between genes and their regulators.
The first level is retesting interactions detected with the same experimental approach and reagents to minimize technical false positives. This can be done by retesting individual interactions, or by performing larger, genome-scale experiments multiple times. For example, in ChIP assays, the DNA regions deemed bound by a TF are often confirmed by quantitative PCR of the ChIPped DNA. However, this only confirms that the DNA fragment was precipitated; it is not a retest of the ChIP assay itself. In Y1H assays, interactions retrieved can be confirmed in freshly grown yeast cells containing the 'DNA bait', using, for instance, a TF-encoding clone .
The second level is confirming an interaction with the same assay but using different reagents. Computationally inferred regulatory interactions can be assessed in an independent dataset, or with a different algorithm. ChIP interactions can be confirmed by using multiple antibodies to the same protein [15, 27]. The signal-to-noise ratio can also be improved by including control experiments of samples in which the TF is removed or reduced . In such experiments, DNA regions that are detected by ChIP both in wild-type and TF mutant or RNA interference (RNAi) samples can be considered false positives, and thresholds can be drawn accordingly. Y1H assays use two reporter genes that are integrated into different locations in the yeast genome, and only interactions that result in the activation of both reporters should be considered, as they are basically detected twice, and therefore confirmed. In addition, independent 'DNA bait' strains and 'TF preys' from different clone resources can be used to confirm interactions.
The third level is detecting an interaction with a different assay of the same type. For instance, to validate computationally inferred regulatory interactions one can use RNAi of a TF and examine changes in the expression of its inferred targets in vivo. This approach has recently been used for a subset of regulatory relationships inferred to control the pathogenic response of murine dendritic cells . Physical interactions detected by Y1H assays can be confirmed by ChIP and vice versa. For example, we have confirmed multiple Y1H interactions in Caenorhabditis elegans and Arabidopsis thaliana interactions by ChIP [17, 21]. Additional support for physical interactions identified either by Y1H assays or by ChIP can be obtained by identifying a putative TF binding site within the DNA fragment bound, either using motif prediction algorithms (for example, [28, 29, 30]) or by interrogating large TF binding site datasets (for example, [31, 32, 33, 34]). In Y1H assays, the putative site can be deleted and interactions with the mutant fragment can be examined. Loss of the DNA-TF interaction with the mutant fragment would confirm that the selected binding site is indeed correct .
The fourth level is observing an interaction in a different type of assay, that is, a regulatory interaction is confirmed by a physical interaction or vice versa. An example of how the regulatory effect of physical DNA-TF interactions can be examined in C. elegans is the generation of transgenic animals that express green fluorescent protein (GFP) under the control of the DNA fragment with which the physical interaction was detected, and subjection of these transgenic animals to RNAi of the relevant TF or crossing them into TF mutant animals [19, 20, 36]. An increase in GFP expression following reduction of the TF would indicate that the TF represses gene expression, whereas a decrease in GFP expression following TF reduction would indicate that the TF is an activator. Other methods used to determine target gene expression in mutant/RNAi animals include quantitative RT-PCR and expression profiling (for example, [17, 37]).
The final level is observing that a TF and its target share functional roles in a biological process, or confer similar phenotypes when mutated and/or overexpressed. Such functional similarities can be uncovered by performing phenotypic experiments, or alternatively, correlations between functions can be investigated in silico using Gene Ontology  and KEGG databases . It is likely that a complete correlation between a TF and its target genes in either a biological process or a phenotype occurs only in a minority of cases because networks are highly interconnected, and TFs often have multiple functions according to developmental, physiological or environmental circumstances.
Limitations of DNA-TF validation: detection limits and interpretation
It is important to note that validation experiments are each subject to their own limitations and that a negative result need not invalidate the original interaction observed. For instance, when RNAi is used to examine regulatory relationships between a gene and its putative regulators, off-target effects can complicate the interpretation. Therefore, it is desirable to observe the same effect on target gene expression with multiple small interfering RNAs (siRNAs) designed to target the same regulator. In C. elegans, not every tissue is equally amenable to RNAi; for instance, most neurons are largely refractory, and interactions occurring there would wrongly be deemed not valid when analyzed by RNAi [40, 41]. When validating Y1H interactions by ChIP, it is important to perform the experiment using samples that not only express the TF of interest, but also under conditions where the TF is active (Figure 1). An example that illustrates this concept is C. elegans DAF-16, a forkhead TF that is located in the cytoplasm in wild-type animals unless they are exposed to nutritional or environmental stress. In daf-2 mutants, however, DAF-16 aberrantly translocates to the nucleus and is constitutively active, and the daf-2 mutant background has therefore been used to identify an initial set of DAF-16-bound regions . Some other TFs could truly be bound to DNA sites in vivo, but be functionally dormant until activation by a ligand or signaling pathway, and regulatory effects would only be detectable under activation conditions. Prior knowledge of TF expression [42, 43] or activity will greatly facilitate the design of an appropriate in vivo experiment. For instance, we focused on animals in the dauer stage (in which their development is arrested) to validate interactions between microRNA promoters and DAF-3, whose expression greatly increases in the dauer stage and which confers a dauer-related phenotype . Finally, some TFs may affect the expression of their target genes in a tissue-specific manner, and some TFs can even function as both an activator and a repressor, depending on the circumstances. For instance, using a transgenic GFP approach, we found that the C. elegans nuclear hormone receptor NHR-45 activates the promoter of nhr-178 in some tissues and under some conditions, whereas it represses it in other tissues under other conditions . This example highlights the complexity of gene regulation and the caution one should take when interpreting 'whole animal' validation assays, such as quantitative RT-PCR.
Limitations of DNA-TF validation
In the past few years, it has become clear that many physical interactions - for instance, those identified by ChIP - do not convey a detectable regulatory consequence on their predicted target genes (for example, [44, 45]). This could be because the regulatory consequences were tested under conditions in which the TF is not active. Alternatively, the wrong gene could have been attributed to the binding event, or the loss or reduction of individual TFs could have been masked by TF combinatorics or redundancy.
Physical interactions between TFs and genomic DNA fragments are often inferred to affect the gene that is in closest linear proximity to the binding event. For interactions that occur close to a transcription start site, either in a gene promoter or just downstream in the transcribed region, this is a reasonable assumption. Interactions involving more distant DNA elements (such as enhancers), however, may not necessarily affect a nearby gene because the genome is not organized as a linear polymer, but rather is organized in three dimensions . Studies using chromatin conformation capture techniques  are providing insights into which genomic regions contact each other to form loops, potentially to bring together enhancers and gene promoters. Integrating physical and regulatory interaction data with structural genome and DNA looping data will facilitate the further dissection of GRNs.
When is an interaction between a genomic DNA fragment and a TF biologically meaningful? When the gene located closest to the binding event changes in expression following loss or reduction of the TF? When the gene and the TF have the same phenotype? Clearly, the answers are not simple, and the assays used to validate interactions are not foolproof. However, when the methods used for GRN delineation are robust and technically sound, individual interactions should not be discarded when one or more validation methods fail to detect regulatory or phenotypic consequences, particularly in complex, whole organisms. Rather, multiple methods need to be combined into increasingly integrated networks to attribute different degrees of confidence to each of the interactions observed.
I would like to thank the members of my laboratory, particularly John Reece-Hoyes, Emma Watson and Lesley MacNeil, as well as Job Dekker and Marc Vidal for discussions. The Walhout laboratory is supported by grants from the National Institutes of Health DK068429 and GM082971, and by the Ellison Medical Foundation.
- 10.Amit I, Garber M, Chevrier N, Leite AP, Donner Y, Eisenhaure T, Guttman M, Grenier JK, Li W, Zuk O, Schubert LA, Birditt B, Shay T, Goren A, Zhang X, Smith Z, Deering R, McDonald RC, Cabili M, Bernstein BE, Rinn JL, Meissner A, Root DE, Hacohen N, Regev A: Unbiased reconstruction of a mammalian transcriptional network mediating pathogen responses. Science. 2009, 326: 257-263. 10.1126/science.1179050.PubMedPubMedCentralCrossRefGoogle Scholar
- 11.Harbison CT, Gordon DB, Lee TI, Rinaldi NJ, Macisaac KD, Danford TW, Hannett NM, Tagne JB, Reynolds DB, Yoo J, Jennings EG, Zeitlinger J, Pokholok DK, Kellis M, Rolfe PA, Takusagawa KT, Lander ES, Gifford DK, Fraenkel E, Young RA: Transcriptional regulatory code of a eukaryotic genome. Nature. 2004, 431: 99-104. 10.1038/nature02800.PubMedPubMedCentralCrossRefGoogle Scholar
- 13.Odom DT, Dowell RD, Jacobsen ES, Nekludova L, Rolfe PA, Danford TW, Gifford DK, Fraenkel E, Bell GI, Young RA: Core transcriptional regulatory circuitry in human hepatocytes. Mol Syst Biol. 2006, 2: 10.1038/msb4100059. 2006.0017Google Scholar
- 14.Zeitlinger J, Zinzen RP, Stark A, Kellis M, Zhang H, Young RA, Levine M: Whole-genome ChIP-chip analysis of Dorsal, Twist, and Snail suggests integration of diverse patterning processes in the Drosophila embryo. Genes Dev. 2007, 21: 385-390. 10.1101/gad.1509607.PubMedPubMedCentralCrossRefGoogle Scholar
- 17.Deplancke B, Mukhopadhyay A, Ao W, Elewa AM, Grove CA, Martinez NJ, Sequerra R, Doucette-Stamm L, Reece-Hoyes JS, Hope IA, Tissenbaum HA, Mango SE, Walhout AJ: A gene-centered C. elegans protein-DNA interaction network. Cell. 2006, 125: 1193-1205. 10.1016/j.cell.2006.04.038.PubMedCrossRefGoogle Scholar
- 18.Vermeirssen V, Barrasa MI, Hidalgo C, Babon JAB, Sequerra R, Doucette-Stam L, Barabasi AL, Walhout AJM: Transcription factor modularity in a gene-centered C. elegans core neuronal protein-DNA interaction network. Genome Res. 2007, 17: 1061-1071. 10.1101/gr.6148107.PubMedPubMedCentralCrossRefGoogle Scholar
- 19.Martinez NJ, Ow MC, Barrasa MI, Hammell M, Sequerra R, Doucette-Stamm L, Roth FP, Ambros V, Walhout AJM: A C. elegans genome-scale microRNA network contains composite feedback motifs with high flux capacity. Genes Dev. 2008, 22: 2535-2549. 10.1101/gad.1678608.PubMedPubMedCentralCrossRefGoogle Scholar
- 21.Brady SM, Zhang L, Megraw M, Martinez NJ, Jiang E, Yi CS, Liu W, Zeng A, Taylor-Teeples M, Kim D, Ahnert S, Ohler U, Ware D, Walhout AJ, Benfey PN: A stele-enriched gene regulatory network in the Arabidopsis root. Mol Syst Biol. 2011, 7: 459-10.1038/msb.2010.114.PubMedPubMedCentralCrossRefGoogle Scholar
- 23.Vermeirssen V, Deplancke B, Barrasa MI, Reece-Hoyes JS, Arda HE, Grove CA, Martinez NJ, Sequerra R, Doucette-Stamm L, Brent MR, Walhout AJ: Matrix and Steiner-triple-system smart pooling assays for high-performance transcription regulatory network mapping. Nat Methods. 2007, 4: 659-664. 10.1038/nmeth1063.PubMedCrossRefGoogle Scholar
- 24.Reboul J, Vaglio P, Rual JF, Lamesch P, Martinez M, Armstrong CM, Li S, Jacotot L, Bertin N, Janky R, Moore T, Hudson JR, Hartley JL, Brasch MA, Vandenhaute J, Boulton S, Endress GA, Jenna S, Chevet E, Papasotiropoulos V, Tolias PP, Ptacek J, Snyder M, Huang R, Chance MR, Lee H, Doucette-Stamm L, Hill DE, Vidal M: C. elegans ORFeome version 1.1: experimental verification of the genome annotation and resource for proteome-scale protein expression. Nat Genet. 2003, 34: 35-41. 10.1038/ng1140.PubMedCrossRefGoogle Scholar
- 25.Rual JF, Hirozane-Kishikawa T, Hao T, Bertin N, Li S, Dricot A, Li N, Rosenberg J, Lamesch P, Vidalain PO, Clingingsmith TR, Hartley JL, Esposito D, Cheo D, Moore T, Simmons B, Sequerra R, Bosak S, Doucette-Stamm L, Le Peuch C, Vandenhaute J, Cusick ME, Albala JS, Hill DE, Vidal M: Human ORFeome version 1.1: a platform for reverse proteomics. Genome Res. 2004, 14: 2128-2135. 10.1101/gr.2973604.PubMedPubMedCentralCrossRefGoogle Scholar
- 26.Deplancke B, Vermeirssen V, Arda HE, Martinez NJ, Walhout AJ: Gateway-compatible yeast one-hybrid screens. Cold Spring Harb Protoc. 2006, 14: 2093-2101. 10.1101/pdb.prot4590.Google Scholar
- 33.Badis G, Berger MF, Philippakis AA, Talukder S, Gehrke AR, Jaeger SA, Chan ET, Metzler G, Vedenko A, Chen X, Kuznetsov H, Wang CF, Coburn D, Newburger DE, Morris Q, Hughes TR, Bulyk ML: Diversity and complexity in DNA recognition by transcription factors. Science. 2009, 324: 1720-1723. 10.1126/science.1162327.PubMedPubMedCentralCrossRefGoogle Scholar
- 34.Matys V, Kel-Margoulis OV, Fricke E, Liebich I, Land S, Barre-Dirrie A, Reuter I, Chekmenev D, Krull M, Hornischer K, Voss N, Stegmaier P, Lewicki-Potapov B, Saxel H, Kel AE, Wingender E: TRANSFAC and its module TRANSCompel: transcriptional gene regulation in eukaryotes. Nucleic Acids Res. 2006, 34: D108-D110. 10.1093/nar/gkj143.PubMedPubMedCentralCrossRefGoogle Scholar
- 35.Reece-Hoyes JS, Deplancke B, Barrasa MI, Hatzold J, Smit RB, Arda HE, Pope PA, Gaudet J, Conradt B, Walhout AJ: The C. elegans Snail homolog CES-1 can activate gene expression in vivo and share targets with bHLH transcription factors. Nucleic Acids Res. 2009, 37: 3689-3698. 10.1093/nar/gkp232.PubMedPubMedCentralCrossRefGoogle Scholar
- 36.Ow MC, Martinez NJ, Olsen P, Silverman S, Barrasa MI, Conradt B, Walhout AJM, Ambros VR: The FLYWCH transcription factors FLH-1, FLH-2 and FLH-3 repress embryonic expression of microRNA genes in C. elegans. Genes Dev. 2008, 22: 2520-2534. 10.1101/gad.1678808.PubMedPubMedCentralCrossRefGoogle Scholar
- 38.Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, Davis AP, Dolinski K, Dwight SS, Eppig JT, Harris MA, Hill DP, Issel-Tarver L, Kasarskis A, Lewis S, Matese JC, Richardson JE, Ringwald M, Rubin GM, Sherlock G: Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet. 2000, 25: 25-29. 10.1038/75556.PubMedPubMedCentralCrossRefGoogle Scholar
- 43.Ravasi T, Suzuki H, Cannistraci CV, Katayama S, Bajic VB, Tan K, Akalin A, Schmeier S, Kanamori-Katayama M, Bertin N, Carninci P, Daub CO, Forrest AR, Gough J, Grimmond S, Han JH, Hashimoto T, Hide W, Hofmann O, Kamburov A, Kaur M, Kawaji H, Kubosaki A, Lassmann T, van Nimwegen E, MacPherson CR, Ogawa C, Radovanovic A, Schwartz A, Teasdale RD, et al: An atlas of combinatorial transcriptional regulation in mouse and man. Cell. 2010, 140: 744-752. 10.1016/j.cell.2010.01.044.PubMedCrossRefGoogle Scholar
- 44.Li XY, MacArthur S, Bourgon R, Nix D, Pollard DA, Iyer VN, Hechmer A, Simirenko L, Stapleton M, Luengo Hendriks CL, Chu HC, Ogawa N, Inwood W, Sementchenko V, Beaton A, Weiszmann R, Celniker SE, Knowles DW, Gingeras T, Speed TP, Eisen MB, Biggin MD: Transcription factors bind thousands of active an inactive regions in the Drosophila blastoderm. PLoS Biol. 2008, 6: e27-10.1371/journal.pbio.0060027.PubMedPubMedCentralCrossRefGoogle Scholar
- 45.Cao Y, Yao Z, Sarkar D, Lawrence M, Sanchez GJ, Parker MH, MacQuarrie KL, Davison J, Morgan MT, Ruzzo WL, Gentleman RC, Tapscott SJ: Genome-wide MyoD binding in skeletal muscle cells: a potential for broad cellular reprogramming. Dev Cell. 2010, 18: 662-674. 10.1016/j.devcel.2010.02.014.PubMedPubMedCentralCrossRefGoogle Scholar
- 47.MacNeil L, Walhout AJM: Gene regulatory networks and the role of robustness and stochasticity in the control of gene expression. Genome Res. 2011, 10.1101/gr.097378.109.Google Scholar