“…some parts of the IPCC process resembled a Soviet-style trial, in which the facts are predetermined, and ideological purity trumps technical and scientific rigor.” – Senator James Inhofe, chairman of the United States Senate Committee on Environment and Public Works

1 Introduction

Although there is a strong consensus regarding anthropogenic climate change among actively publishing climate scientists (Anderegg 2010; Anderegg et al. 2010; Bray 2010; Cook et al. 2013; Doran and Zimmerman 2009; Rosenberg et al. 2010), there is still much debate in the media and among the public (Boykoff and Boykoff 2004, 2007; Dunlap and McCright 2011; Oreskes and Conway 2010; Zehr 2000). Much of the opposition to climate change exists in the context of political rather than scientific discourse and the link between climate change skepticism and explicitly political bodies such as conservative think tanks is direct and uncontroversial (Dunlap and Jacques 2013; Dunlap and McCright 2011; Jacques et al. 2008; McCright and Dunlap 2003). Nonetheless, there are some trained scientists who question anthropogenic climate change on apparently scientific grounds, though typically under the auspices of think tanks. The most striking example of this comes from the Heartland Institute (a conservative think tank), which formed a Nongovernmental International Panel on Climate Change (NIPCC) as an alternative to the Intergovernmental Panel on Climate Change (IPCC). In 2013, the NIPCC authored Climate Change Reconsidered II: Physical Science (hereafter referred to as ‘NIPCC’; Idso et al. 2013), a scientific report that is a direct response to IPCC’s Working Group 1: The Physical Science Basis (hereafter referred to as ‘IPCC’; Stocker et al. 2013), also published in 2013. NIPCC has published previous reports as well (Singer 2008; Idso and Singer 2009; Idso et al. 2011). Here we focus entirely on the most up to date publication under the assumption that it is likely the best and most comprehensive example of scientific writing from NIPCC and, perhaps, climate change skeptics as a whole.

The NIPCC report is a unique element in the politicization of climate science. Two of its three lead authors, Craig D. Idso and S. Fred Singer, have doctorate degrees in physical sciences and the report boasts “nearly 50” additional authors (see supplementary materials for a breakdown of NIPCC authors). The report itself is 993 pages long and highly technical. Moreover, a comparison of older NIPCC (Idso and Singer 2009; Climate Change Reconsidered I) and IPCC (Solomon et al. 2007; Climate Change 2007: The Physical Science Basis) reports revealed that the two groups cited the same scientific journals as their most important sources (Jankó et al. 2014). It is very unlikely that non-experts (present authors included) would have the requisite knowledge to be able to distinguish the NIPCC and IPCC reports based on the validity of their scientific arguments. Both reports explicitly claim that they represent comprehensive assessments of climate change science. Indeed, both reports are superficially ‘scientific’ and the NIPCC report goes out of its way to appear like the IPCC report (even by having a similar name). Thus, if taken at face value, the reports should follow similar discourse patterns (Biber and Gray 2013; Robinson et al. 2008; Zeiger 1999). However, the reports were not written in a political vacuum: the politicization of climate science should be evident not only in the language content (i.e., what is said), but also the language style (i.e., how it is said) of the many authors of the competing reports. The current paper investigates whether the two are in fact the same stylistically or not.

1.1 Hypotheses

A common tactic among climate change skeptics over the past two decades has been to accuse climate scientists of being “alarmist”. Indeed, this is mentioned explicitly in the NIPCC report (e.g., Foreword, p. vi). However, at least in terms of climate change predictions, IPCC scientists have actually been conservative; routinely under-predicting the impacts of climate change (Brysse et al. 2013; Freudenberg and Muselli 2013). Although scientists in general are conservative when it comes to predictions (to avoid Type 1 errors), the politicization of climate science may be at least partially responsible for IPCC conservatism (Anderegg et al. 2014; Freudenberg and Muselli 2010). Specifically, the charged political atmosphere surrounding climate change research may bias IPCC authors toward caution and tentativeness for fear (either implicit or explicit) of controversy.

The perspective of the NIPCC reports’ authors is very different. NIPCC’s goal is to present its case by actively refuting and discrediting the IPCC. Since very little of the published scientific research contains arguments against the anthropogenic climate change consensus (see Cook et al. 2013; Oreskes 2004), skeptics are in a position where they must reinterpret published work. This reactive position should lead to the opposite pattern to the one found in the IPCC report. That is, the NIPCC authors should be less conservative (i.e., more committed to the certainty of their estimates) than their IPCC counterparts because they have the more difficult task of usurping an established consensus.

Alternatively, despite being conservative in their predictions, the overwhelming consensus on anthropogenic climate change could potentially lead IPCC authors to use a less cautious language style than the NIPCC authors. Although this might be justified given the scientific context of the respective documents (i.e., more evidence warrants more certainty), it would nonetheless be consistent with (though not strong evidence for) the myriad claims that IPCC climate scientists are ‘alarmist’ and perhaps even less scientifically rigorous than prominent skeptics. In the current study, we employ a set of computational text analyses to explore the lexical features of the two climate change reports.

1.2 Predictions

Previous research has demonstrated that lack of absolute certainty in science texts is saliently represented by the frequent use of hedging or more tentative language (e.g., the use of words such as possible, probable, might; Adams Smith 1984; Fløttum and Dahl 2012; Gosden 1993; Hyland 1998). Moreover, reduced explicitness in science texts is achieved by the use of nominalizations (Biber and Gray 2010; Halliday 1979) and passive constructions (Halliday and Martin 1993). These linguistic devices reduce explicitness because they omit certain elements such as the agent. For instance, the active sentence, “The committee observed temperature change” can turn into the passive form, “Temperature change was observed” in which the agent (the committee who observed temperature change) has been omitted. Similarly, the nominalized construction, “Temperature change observation,” in which the verb ‘observe’ was transformed into the noun ‘observation’, omits the agent. Scientists generally use less explicit language rather than absolute claims, in part because their claims are inferential and might need to be reinterpreted based on future research (Hyland 2005). In other words “hedges are disclaimers” (McNamara et al. 2014, p. 186). As a consequence, we expect the use of these language forms to be more evident in the IPCC report than the NIPCC report.

Moreover, the NIPCC authors explicitly assume the position of a “Red Team” (NIPCC, pp. vi, 1) whose goal is to counter the arguments provided by the “Green Team” (in this case IPCC). NIPCC is thus in a reactive position, with the goal of actively refuting the IPCC report. In this context it is instructive to consider how this position might have affected the scientific discourse of the report. Indeed, there is some linguistic evidence of a less “formal” language style early on in the NIPCC report when the IPCC report is described as “preposterous … alarmist and incompetent” (p. 15), but also accused of “trumpeting catastrophes” (p. 15). While this kind of language would normally be considered inappropriate (i.e., too informal) in the context of scientific writing, here it clearly serves the purpose of aggressively disputing the credibility of the IPCC report. In light of this we investigate whether the NIPCC report uses less neutral language (i.e., more emotional words; e.g., alarmist, wrong), but we also consider other measures related to formal/informal writing styles. For example, a more formal style (characteristic of scientific writing) is expected to: be nominal (i.e., feature more frequent use of nouns; Biber and Gray 2013), include more difficult (e.g., abstract) words, have more complex syntax, and include increased cohesion (e.g., Graesser et al. 2014). Thus if the NIPCC report indeed contains a less formal style compared to the IPCC report, then it should contain lower noun incidence, more frequent words, more concrete words, simpler syntactic structure, and less cohesion than the IPCC report.

2 Method

2.1 Tool description

The climate change reports were analyzed using the Coh-Metrix text analyzer (Graesser et al. 2011; McNamara et al. 2014), Linguistic Inquiry and Word Count (LIWC; Tausczik and Pennebaker 2010), and AntConc 3.3.5 concordancer analysis toolkit (Anthony 2012).

Coh-Metrix analyzes texts on various levels of language and discourse. We used Coh-Metrix to determine number of modifiers per noun-phrase (as an index of nominalization, Lambert and Robinson 2014; but also note that noun modification makes text more dense and abstract; Biber and Gray 2010; Parkinson and Musgrave 2014), incidence of nouns, incidence of passive voice, word frequency (all words), word concreteness, words before main verb (as an indicator of syntactic complexity, since an increase in number of words before the main verb increases syntactic complexity; e.g., Crossley and McNamara 2011), and the Measure of Textual Lexical Diversity (MTLD; McCarthy and Jarvis 2010) as a text cohesion indicator (e.g., a lower MTLD value indicates higher cohesion or more word repetition across the text; McNamara et al. 2014).

LIWC analyzer compares words in the text against a set of words categorized into over 70 language dimensions. Here, LIWC was used to extract the information about the proportions of tentative words and emotional (we aggregated proportion of positive and negative emotion) words. In addition, the proportions of emotional words were determined from concordance lines.

AntConc analyzer contains a set of tools (e.g., concordancer, keyword frequency generators). We used AntConc to perform concordance analysis. AntConc concordancer displays a searched word in context. In other words, concordancer lists the searched word together with a text that surrounds it (the amount of surrounding text is specified by researcher). This allows us to determine the context in which words or terms are commonly used in text.

2.2 Corpus

Our corpus consisted of two climate change reports: “Working Group 1: The Physical Science Basis” (IPCC) and “Climate Change Reconsidered II: Physical Science” (NIPCC). The reports were freely available online at http://www.ipcc.ch/report/ar5/wg1/ (IPCC) and http://www.nipccreport.org/reports/ccr2a/ccr2physicalscience.html (NIPCC). The reports were both downloaded on April 4th, 2014. All headers and footers, figures, figure captions, boxes, appendices, and references were manually removed before the analyses. All lexical indices were computationally extracted.

We analyzed the reports in their entirety (i.e., Chapters 1–14 of the IPCC report, and Chapters 1–7 of the NIPCC report). Because of the length of the reports, we split them into files containing approximately 20,000 words (M = 20321, SD = 1135), and then averaged the results in the Coh-Metrix analysis. Overall, over 995,000 words were analyzed.

3 Results

To determine the level of certainty in the language style of the two reports, we measure hedging (i.e., more tentative language), nominalizations, and passive constructions. The use of these language forms was more evident in the IPCC report (Table 1). This indicates that, as expected, the IPCC authors used more conservative (i.e., more cautious, less explicit) language to present their claims compared to the authors of the NIPCC report.

Table 1 Coh-Metrix and LIWC scores for the language measures in the IPCC (Working Group 1: The Physical Science Basis, 2013) and NIPCC (Climate Change Reconsidered II: Physical Science, 2013) reports compared to meaningful references
Full size table

Also, as predicted, the NIPCC report contained less “formal” language than the IPCC report. Specifically, NIPCC used more emotional words, but also had lower noun incidence, used more frequent words, more concrete words, had simpler syntactic structure, and lower cohesion than the IPCC report.

Given that the NIPCC was a response to the IPCC, we also employed a concordance analysis which isolated in-text references of the term “IPCC” in the NIPCC report (search query was set to specifically extract only the string IPCC with zero or more characters following, e.g., IPCC, IPCC’s, IPCC-, but without including strings with preceding characters, e.g., NIPCC). The analysis revealed 515 in-text references of the term “IPCC” in the NIPCC report. Moreover, mentions of the term “IPCC” were distributed across the report, another indicator of NIPCC’s reactive position and systematic attempt to discredit the IPCC report (see Fig. 1). Next, we analyzed concordance lines to determine the presence of emotional (positive + negative) words around the term “IPCC” (each line contained 100 characters on either side of the term). The results demonstrated that 3.02 % of the words in the concordance lines were emotional relative to a base-rate of 2.71 % in the whole text. Thus, not only was the language used by the NIPCC authors more emotional than that used by the IPCC, but emotional language was likely used in the NIPCC report to directly challenge and discredit the IPCC.

Fig. 1
figure 1

Concordance Plot. The relative position of each hit (i.e., “IPCC”) in the NIPCC report is displayed as a line in bar chart (515 hits)

Full size image

In summary, the NIPCC authors appeared to be more certain in their language than the IPCC authors, while the authors of the IPCC report appeared to use a more formal and more neutral language style than the NIPCC report. The results were consistent with our predictions on every predetermined measure intended to reflect differences in language use among NIPCC and IPCC authors. This indicates considerable differences between the reports in terms of scope. However, some of these differences may seem small and, therefore, it is necessary to determine their magnitude relative to meaningful reference points.

The authors of the Coh-Metrix text analyzer provide lexical indices norms for science texts across grade levels (McNamara et al. 2014). These norms were created by analyzing a subset of a large corpus of texts (The Touchstone Applied Science Associates (TASA); Landauer et al. 1998) containing 37,651 texts and over 11 million words. In general, science text formality increases with increased grade level (Graesser et al. 2014). Thus, we calculated the mean difference between the six consecutive grade groups (i.e., K to 1, 2 to 3, 4 to 5, 6 to 8, 9 to 10, and 11 to College) on each respective measure, but also the difference between the two highest grade groups (i.e., grades 9 to 10, and grades 11 and higher) and used these measures as meaningful reference points (Table 1). This analysis clearly demonstrates that comparably sized differences (to the majority of our results) are found when contrasting science texts among consecutive grade levels.

There are no LIWC norms for science text that we are aware of. However, Hyland (1998) reported that hedging represented more than 2 words in every 100 words in research articles published in leading journals in cell and molecular biology (in our case, the IPCC report contains 2.11 tentative words in 100 words, while the NIPCC contains 1.72 tentative words in 100 words). Although this comparison should be treated with caution as it crosses disciplines and report types, it nonetheless converges on the same conclusion as the results using the Coh-Metrix text analyzer.

4 Discussion

Language use depends largely on context (e.g., Biber and Conrad 2009) and the debate surrounding climate change is no exception. We found substantial differences between the IPCC and NIPCC reports despite the fact that they were both intended to be comprehensive assessments of climate science research and were each authored by many individuals. On most measures, differences between the reports were larger or equivalent to differences between entire grade levels in science textbooks. Although there are surely many factors that determine which words scientists decide to use, our results are consistent with the idea that political context is an important factor for science communication.

Previous research indicates that IPCC authors have been conservative in the content of their predictions (Brysse et al. 2013; Freudenberg and Muselli 2013). To this end, one of the key results of our analysis was that the IPCC authors used more tentative language. Climate change proponents such as the IPCC authors are surely aware of the potential cost of making an error (Lewandowsky et al. 2015), and this awareness (either implicit or explicit) may have subtle effects on their language style relative to climate change skeptics. The public debate on climate change may be “seeping” into the scientific process, causing a number of downstream effects (for a discussion of “seepage”, see Lewandowsky et al. 2015).

It should be noted, nonetheless, that the present analysis does not provide direct evidence that IPCC climate scientists use a particularly conservative language style relative to other mainstream scientists. This would require a third comprehensive report on a topic in an area that has not been heavily politicized. However, even in such a case, it would be difficult to know if the differences were due to the change in topic or the change in relative politicization. At any rate, our findings are consistent with research indicating conservatism in predictions among IPCC climate scientists (Brysse et al. 2013; Freudenberg and Muselli 2013) and, at the very least, fly in the face of claims that IPCC scientists are ‘alarmist’ and less scientifically rigorous than climate skeptics.

The NIPCC report itself is an example of why climate scientists may be concerned about potentially overstating the case for climate change. NIPCC authors attempted to actively refute the IPCC throughout their report. Although this is unsurprising given that the NIPCC emerged as a response to the IPCC, the more important result of our analysis was the less formal tone of this refutation. This occurred despite the NIPCC report representing perhaps the most scientific and comprehensive from the perspective of climate change skeptics. This has important implications for the denialist movement as the NIPCC report represents a key example that may be used by skeptics to argue against the climate change consensus in the public sphere – a major problem for proponents because the belief that scientists disagree about climate change is a major roadblock to support for changes in climate policy among the public (Ding et al. 2011; McCright et al. 2013; van der Linden et al. 2015). We hasten to add, however, that our language analysis does not bear on the accuracy of the claims within the IPCC or NIPCC documents. It is quite possible for a less linguistically formal scientific document to be ultimately more accurate. Rather, the reported differences in language use should be thought of as reflective of the different perspectives of the authors of the two reports.

Our results indicate that politics may have important implications for not only what is said in scientific discourse, but how it is said. Moreover, the language style used by climate change skeptics suggests that the arguments put forth by these groups warrant skepticism in that they are relatively less focused upon the propagation of evidence and more intent on discrediting the opposing perspective.