Editors’ Preface: Trends in experimental economics (1975–2018)

1 Goodbye!

The first issue of the Journal of the Economic Science Association was published in July 2015. Our editorial for that issue started with a “Welcome!” After 5 years as Co-Editors, the time has come to “pass on the torch” to a new set of editors. With this comes the inevitable desire to reflect on what has been achieved in this period, and the challenges that may lie ahead for our Association.

In the preface of the first issue, we explained the need that led to the establishment of JESA as follows:

“The main reason for establishing JESA is that experimental findings now feature prominently in leading journals, garner hundreds (sometimes thousands) of citations and influence economic theory and public policy. With this comes the need to evaluate the robustness as well as the internal and external validity of conclusions drawn from experimental data more carefully and extensively.” (Nikiforakis and Slonim 2015, p. 4).

To back up this claim, among others, we presented data showing the evolution of experimental research published in the “top-5” economics journals (American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies) between 1975 and 2014. The data illustrated the explosion in the number of experimental papers published during this 40-year period. However, they also captured an unexpected decline after 2010. Although the decline was substantial, it was not the first time a drop in the number of experimental papers published in the top-5 was observed. This led us to make the following comment:

“[The evidence] also shows a drop in the number of lab experiments published in the past 4 years. The drop is due to a reduction in papers published in the American Economic Review. There are too few observations to determine whether this is a temporary drop such as that observed during 1991–1996 or a permanent drop.” (Nikiforakis and Slonim 2015, p. 3).

Five years have passed since then, during which we oversaw the publication of ten issues for JESA. That may not sound like much, but it surely felt like it was many more. For our farewell editorial, we decided to explore whether the drop in experimental papers published in the top-5 was temporary or an early-warning signal of things to come.

2 Recent trends in experimental economics

We extend the prior analysis we did in Nikiforakis and Slonim (2015).¹ Following Card et al. (2011), lab experiments combine experiments conducted in standard laboratories, lab-in-field experiments, as well as online experiments using standard experimental techniques. Field experiments include any study that generates original data, where the data collection happens in an environment that exists naturally (i.e., the authors do not set up or use a laboratory), and where the authors themselves explicitly choose one or more aspects of the environment to manipulate, and this manipulation happens in a fully controlled and randomized way. Randomized control trials (RCTs) are included in this category.

Figure 1 graphs the number of experimental economic papers published between 1975 and 2018 in the aforementioned top-5 general-interest economics journals. The figure captures two main trends: (1) the explosive growth in the period 1975–2010, and (2) the sharp decline since 2010. As the period of expansion was already discussed in Nikiforakis and Slonim (2015), our focus will be on the period 2011–2018.

Fig. 1

Experimental papers published in the “top-5” economics journals (1975–2018)

Figure 1 clearly shows that the drop in “lab papers” published in top-5 economics journals (i.e., papers reporting data from lab experiments) that started after 2010 has persisted. From the peak in 2009–2010 to 2017–2018, there was a 66% drop in “lab papers”. In fact, the number of “lab papers” has experienced a monotonic decline every year during this period. The period 2017–2018 saw the lowest number of “lab papers” published in the top-5 in 20 years, i.e., since 1997–1998. Unless this pattern is reversed in 2019, the “’10s” will have to go down in the history of experimental economics as a “dark” decade, when the gains from the “’00s” were erased.

As can also be seen in Fig. 1, the number of field experiments published in the top-5 has been steadily growing since 2000. However, again, we observe a small drop in the last 4 years, starting in 2015. This surprised us given the broad classification of studies as “field experiments” and the growing popularity of RCTs, which are included in that category. As a result, the total number of all papers reporting data from controlled experiments has dropped by 37% since 2014. This is a substantial decline. Given that many departments across the globe place significant weight on top-5 publications when making tenure decisions, this development raises concerns, not only about the academic career of untenured experimentalists, but also for economics as a scientific field, as envisioned by the founders of the Economic Science Association.

To better understand this development, we conducted additional analyses. Figure 2 graphs the total number of papers published in the top-5 between 1975 and 2018. Between 2011 and 2018, there has been a 10.5% increase in the number of papers published in the top 5. Therefore, the decline in the number of experimental papers cannot be attributed to a reduction in the overall number of papers published in the most prestigious journals in economics.

Fig. 2

Total number of papers published in the “top-5” between 1975 and 2018

In Nikiforakis and Slonim (2015), we reported that the decline which started in 2011 could be attributed to the reduction in experimental papers published in the AER. This coincided with the change in the journal’s editorial board. A natural question, therefore, to ask is what fraction of experimental papers does each of the five journals currently publish (2015–2018). As can be seen in Fig. 3, with the exception of Econometrica, the other journals give a similar share of their pages to experimental papers. AER, JPE and QJE publish a noticeably higher fraction of field than lab experiments. Econometrica is the only journal that gives a higher share of its pages to lab rather than field experiments.

Fig. 3

Experimental papers as a fraction of all papers published in the “top-5” (2015–2018)

Although less of a concern for the purposes of this analysis, before one rushes to use the data in Fig. 3 to devise publication strategies, it might be useful to note that the five journals publish different quantities of papers. Figure 4 illustrates that 38% of all papers published in the top-5 were in the AER. Similarly, 39% of all experimental papers published in the top-5 were in the AER.

Fig. 4

Share of papers published in the “top-5” by journal in 2015–2018

The patterns seen in Figs. 1 and 3 are particularly worrisome when one takes into consideration the fact that the number of experimental economists appears to have continued growing during the “’10s”. Specifically, according to the official ESA records, between 2011 and 2018, there was a 34.5% increase in the number of ESA members. The ESA through its Executive Committee is already taking steps to explore more deeply the causes for the sharp decline observed in the past 8 years.

3 Final remarks

In 1973, Paul McCartney sang: “What does it matter to you/when you got a job to do/you got to do it well/you got to give the other fellow hell” (McCartney 1973). And so, despite these negative developments in our field, we press on.²

For the two of us, the past 5 years have been quite unique. Almost on a daily basis, we would exchange emails with each other and spoke frequently over Skype. Most of the time, we would give each other feedback on a paper that was under consideration or a decision letter that was about to be sent out to authors.³ Sometimes, we would discuss lofty matters such as best-practices in experimental economics and what the standards for publishing a paper at JESA should be. Other times, we would converse about more mundane topics such as dealing with a (mal)function of the online platform. What kept us going through it all was our unwavering desire to establish JESA as an important outlet for academic research, and help further advance the standards of experimental research in economics. Time will tell how successful we were.

We did our best to be fair and helpful, even when the decisions we were making were not the ones the authors would like. We are grateful to many people: everyone who submitted their papers to JESA; David Cooper, Ian Krajbich and Charles Noussair for editing the Special Issue on Choice-Process Data; the co-editors at Experimental Economics between 2014 and 2019 with whom we exchanged notes and papers on several occasions (David Cooper, John Duffy, Lata Gangadharan, Charles Noussair, Marie Claire Villeval, and Roberto Weber); and, of course, our wonderful editorial board, advisory editors and reviewers for helping us in our mission.

It was a distinct honor to serve as JESA’s inaugural editors. We are grateful to those who entrusted us with this role. We are delighted that, following an extensive search for our successors, two outstanding colleagues—Maria Bigoni and Dirk Engelmann—agreed to be the new co-editors. We wish them all the best.

So, goodbye from us. Here is to the Journal of the Economic Science Association having many more retiring co-editors!

Nikos Nikiforakis

Robert Slonim

Co-editors, Journal of the Economic Science Association

Nov. 30, 2019