What’s more general than a whole population?
Statistical inference is commonly said to be inapplicable to complete population studies, such as censuses, due to the absence of sampling variability. Nevertheless, in recent years, studies of whole populations, e.g., all cases of a certain cancer in a given country, have become more common, and often report p values and confidence intervals regardless of such concerns. With reference to the social science literature, the current paper explores the circumstances under which statistical inference can be meaningful for such studies. It concludes that its use implicitly requires a target population which is wider than the whole population studied — for example future cases, or a supranational geographic region — and that the validity of such statistical analysis depends on the generalizability of the whole to the target population.
KeywordsTarget Population Statistical Inference Acute Leukemia Credible Interval National Cancer Registry
If Czech history could be repeated, we should of course find it desirable to test the other possibility each time and compare the results. Without such an experiment, all considerations of this kind remain a game of hypotheses .
One way to preserve sampling-based statistical inference is to adduce a group that is even more general than the whole population that was studied. This kind of wider group is called here a target population, although the term is sometimes used as a synonym for the sampling population . The target population is the group ‘about which conclusions are to be made’  but, when it differs from the sampling population, it is not subject to the sampling variation formalism. Some authors define target populations to be real  (p361) while for others they do not necessarily exist, at least not yet. For example, Kirkwood and Sterne say that enumeration of a target population may be impossible because it ‘often includes not only all persons living at present but also those that may be alive at some time in the future’  (p10). Despite being currently intangible, such target populations can be important to the application in hand, the cited authors’ example being future recipients of an experimental vaccine. Target populations (in this sense) are sometimes called superpopulations, which comprise ‘all possible persons that ever were or ever could be targets of inference’ .
In the following sections we look at attempts to retain statistical inference in whole population studies, then argue that the problem is better seen as one of generalizability.
The bootstrap generates variation by sampling the original data with replacement. This can actually be carried out in practice, unlike the notional repeated sampling of a population. However the validity of the bootstrap depends on ‘independent identical sampling from an unknown distribution’  so the original data are themselves still assumed to result from a sampling process.
Consider the following two examples of a binary outcome with complete population coverage
Presidential election: there is 100% turnout in a national presidential election, with each vote being either for candidate A or candidate B.
Cancer registry: male or female sex is registered for all cases in a national registry which has 100% coverage of the type of cancer in question.
In the election example, is it meaningful to estimate a sampling error for the proportion voting for candidate A? The answer seems to be clearly ‘no’. This is because the purpose of the election is to choose a president, which is done on the basis of the observed proportion of votes cast. Any kind of interval estimate serves no purpose because there is no generalizability beyond the election.
In the cancer registry example, is it meaningful to estimate a sampling error for the proportion of cases who are female? Some would say ‘no’ on the basis that it’s a complete population enumeration with no sampling error. Similar examples in the literature show that some authors would say ‘yes’. This implies an attempt to generalize beyond the population observed, but what is this wider target population? Conceivably future cases, or a wider, supranational geographical area, although often this is left unspecified.
Representativeness and generalizability
The above reasoning suggests that the validity of either frequentist or Bayesian inference depends on the extent to which the group being studied — whether or not it is a whole population — is representative of the target population. Porta  allows a sample to be representative of a population if it is ‘typical in respect of certain characteristics, however chosen’, i.e., without requiring a particular selection method such as random sampling. Hence a study group can potentially be representative of a given target population, and its findings generalizable to it, even if the target population is not amenable to sampling.
The current paper argues that the problem of inferential methods in whole populations is most usefully understood as one of generalizability (Fig. 2). Thinking in these terms helps show that inferential methods are meaningless for some situations while for others their use is at least arguable (see Table 1).
Lack of information on generalizability has been identified as a limiting factor in the translation of research findings to policy [17, 18, 19]. The CONSORT guidelines for clinical trials , and STROBE for observational studies  mandate discussion of generalizability but give no guidance on how to assess it. Point 21 of STROBE, for example, is ‘Discuss the generalisability (external validity) of the study results’. Several frameworks for assessing generalizability have been proposed [17, 22, 23, 24]. Inferential analysis of whole population studies is justified only as far as such studies are generalizable, yet reporting of generalizability in other kinds of study is often poor [18, 19]. The extent to which this also applies to whole population studies is briefly assessed in the following section.
Literature review on generalizability of whole population studies
A literature review was carried out to assess the extent to which whole population studies assess their own generalizability. The search term ‘whole population [TI]’ was used in PubMed on 25 February 2015, restricted to publications in 1994 or later, yielding 64 publications. They were retained if containing primary data on studies of humans, and reporting p values or confidence intervals in the abstract. The full text of each of the resulting 13 studies was reviewed for description of its generalizability or external validity (Table 2). The populations were typically either whole countries (e.g., Iceland) or subnational administrative regions of different sizes (e.g., Western Australia, Isle of Wight). Only two papers [25, 26] made reference to wider target populations to which the results might be generalized, and which might give meaning to the p values and/or confidence intervals used. These findings can only be suggestive because a limitation of the search is that it does not cover all studies which the current paper would call ‘whole population’. For example, the previously-cited Danish National Acute Leukemia Registry  has estimated coverage of more than 98 %, and a PubMed search for its name yields 11 studies, but none of these met the current search criteria. Among the studies which were identified, however, few are concerned about their generalizability despite the fact that, on the reasoning of the current paper, the validity of their statistical inference depends on it. In particular none of them mentioned the STROBE guidelines which mandate that generalisability be considered.
Assessment of generalizability or external validity
‘We are confident about the generalisability of our analytic findings on associations with stimulant medication use Australia-wide’
North East Scotland
‘Our results could be generalisable to young children across the UK'
‘Top End’ of the Northern Territory of Australia
Isle of Wight, United Kingdom
Isle of Wight, United Kingdom
Whole population studies often calculate p values and confidence intervals which have no explicit theoretical basis. This is a problem that social sciences have, perhaps, confronted more directly than epidemiology [9, 27]. Having concluded that purely statistical workarounds are futile, should we eschew inferential statistics altogether and concentrate instead on descriptive statistics ? The current paper advocates a less absolute position; that the conclusions of whole population studies are valid to the extent that their study populations are representative of a wider target population. Hence the problem is converted into one of generalizability. In turn this should be manifest in each study’s research question. For example, who is intended to benefit from analysis of national cancer registry data to be applied; future cases in the same country, and/or those further afield? If the former, making this explicit may highlight needs for particular statistical methods, such as time series analysis on the existing data disaggregated by time. Some target populations may not be possible to sample, e.g., because they lie in the future. Similarly, causal inference deals with unobserved or counterfactual outcomes [29, 30], and can be cast in terms of target and source populations , and is a promising approach for analysis of whole population studies.
In epidemiology, although aspirations to generalizability are not always met, groundwork has been done on its objective assessment [17, 22, 23, 24, 32]. For inferential statistics of whole population studies to be more meaningful, they should fully comply with existing guidelines on reporting generalizability [20, 21], and these guidelines should themselves be updated to reflect the developing best practice.
I receive support from the United Kingdom Medical Research Council (MRC) and Department for International Development (DFID) (MR/K012126/1). I am grateful to Karim Anaya-Izquierdo for introducing me to the field of generalizability, and to him, Lyda Osorio, and anonymous referees for helpful discussion of an earlier version of this paper.
- 1.Kundera M. The Unbearable Lightness of Being. London: Faber and Faber; 1984.Google Scholar
- 2.Armitage P, Berry G, Matthews JNS. Statistical Methods in Medical Research. 4th ed. Oxford: Blackwell Scientific Publications; 2001.Google Scholar
- 3.Snedecor GW, Cochran WG. Statistical Methods. 8th ed. Ames: Iowa State University Press; 1989.Google Scholar
- 4.Fathalla MF, Fathalla MMF. A Practical Guide for Health Researchers. World Health Organization Regional Office for the Eastern Mediterranean: Cairo; 2004.Google Scholar
- 9.Berk RA, Western B, Weiss RE. Statistical inference for apparent populations. In: Marsden PV, editor. Sociological Methodology 1995. Cambridge: MA: Blackwell Publishers; 1995. p. 421–58.Google Scholar
- 10.Porta M. Dictionary of Epidemiology. 5th ed. Oxford: Oxford University Press; 2008.Google Scholar
- 11.Coggon D, Rose G, Barker DJP. Epidemiology for the Uninitiated. 4th ed. London: BMJ Publishing; 1997.Google Scholar
- 12.Rothman KJ, Greenland S. Modern Epidemiology. 2nd ed. Philadelphia: Lippincott-Raven Publishers; 1998.Google Scholar
- 13.Kirkwood BR, Sterne JAC. Essentials of Medical Statistics. 2nd ed. Oxford: Blackwell Scientific Publications; 2003.Google Scholar
- 14.Bollen KA. Apparent and nonapparent significance tests. In: Marsden PV, editor. Sociological Methodology 1995. Cambridge: MA: Blackwell Publishers; 1995. p. 459–68.Google Scholar
- 17.General Accounting Office. Cross design synthesis: a new strategy for medical effectiveness research. Washington, DC: GAO; 1992.Google Scholar
- 24.Watts P, Phillips G, Petticrew M, Harden A, Renton A. The influence of environmental factors on the generalisability of public health research evidence: physical activity as a worked example. The international journal of behavioral nutrition and physical activity. 2011;8:128.PubMedCrossRefPubMedCentralGoogle Scholar
- 28.Ballinger C. Why inferential statistics are inappropriate for development studies and how the same data can be better used. In: MPRA. University Library of Munich, Germany: Paper 29780; 2011.Google Scholar
- 30.Pearl J. An Introduction to Causal Inference. Int J Biochem. 2010;6(2):7.Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.