Mea culpa I have conducted clinical research using administrative databases. These types of studies have strong advocates, convinced of the value of getting access to very large cohorts, and fierce detractors, who contend that administrative databases contain information that is too limited and unreliable to answer any research question with confidence. I went from skeptic to user when I started to understand what administrative databases could offer and, very importantly, what they cannot.

In this issue of Neurocritical Care, Dasenbrock et al. report their analysis of timing of tracheostomy and a outcomes after aneurysmal subarachnoid hemorrhage using data from the Nationwide Inpatient Sample (NIS) (years 2002–2011) [1]. They found that variation in hospital practices was a strongest predictor of tracheostomy timing than comorbidities or the severity of the hemorrhage. Subarachnoid hemorrhage (SAH) severity was estimated using the previously described NIS-SAH severity scale [2]. Scores in this scale are derived from a weighed calculation of various ICD-9 diagnostic (e.g., coma and hydrocephalus) and procedure codes (e.g., ventriculostomy), and the scale was found to have good concordance with the Hunt and Hess grading system. Dasenbrock et al. also found that later tracheostomy was associated with more pulmonary complications and venous thromboembolism (though only in patients with less severe subarachnoid hemorrhage), but the timing of tracheostomy did not influence in-hospital mortality or disposition at discharge. No information was provided in regard to hospital length of stay or financial costs.

The study by Dasenbrock et al. represents a good example of how a large administrative database, such as the NIS, can be used to try to answer a practical clinical question. Undoubtedly, the study has numerous limitations and has to be interpreted with caution. The NIS-SAH severity scale has not undergone extensive validation, multiple unaccounted factors could have affected the timing of tracheostomy, and functional outcomes cannot be gathered from the NIS. Yet, this study provides a broad perspective on tracheostomy practices across the country and highlights the extensive variations in these practices.

There are many national or regional administrative databases that can be used for medical research, including research in neurocritical care [3]. They afford investigators access to data from very large numbers of hospital admissions. The representation of many hospitals, often from many different geographic locations, generally ensures external validity and also allows for comparison across groups of hospitals or regions. Some of these databases provide financial information that can be used to perform economic analyses. Being longitudinally acquired, administrative databases can be particularly valuable to evaluate temporal trends (for instance in admissions for specific diseases, utilization of treatments, or in-hospital mortality) [4]. Linking databases can be a powerful way of addressing a question, as exemplified by another study recently published in Neurocritical Care, in which Martin et al. acquired data from a sample of Medicare beneficiaries hospitalized for neurological diseases with critical care claims and then linked the National Provider Identifier numbers from those claims to a registry of all United Council for Neurological Subspecialties diplomates in neurocritical care [5]. The conclusion that less than half of neurocritical care services to Medicare beneficiaries are provided by clinicians certified in neurocritical care is both reliable and useful.

However, investigators must be aware of major unavoidable limitations of administrative databases, some of which may completely negate their usefulness to answer the proposed research question. As a general rule, administrative database does not provide the type of granular clinical information that we typically consider necessary to conduct rigorous research. Very often, variables of interest cannot be abstracted from the database. At times, these variables can be replaced by alternative estimations (as exemplified by the NIS-SAH severity scale being used to fill the unavailability of clinical grades in cases of subarachnoid hemorrhage); these replacements may be more or less reliable and should always be regarded cautiously when reaching conclusions. Comorbidities may be available based on diagnostic codes, but their severity is generally not known [6, 7]. Administrative databases are typically restricted to the duration of the hospital admission and do not offer any information after discharge. Therefore, they are not helpful when functional outcome is the endpoint of main interest.

Data accuracy is the most concerning pitfall of administrative registries. Poor documentation in the records and coding errors can threaten the veracity of the data [8]. Diagnostic and procedural codes may be ambiguous or non-specific. Coders may be insufficiently trained in how to interpret complex medical terms and they may miss or misclassify relevant diagnoses when the language in the records does not conform to the terminology of the code [6, 8]. Different registries serve different purposes, and the criteria used to collect the data may vary as a consequence. It is sobering to note that information on the same variables may be substantially discordant across different databases [9, 10]. Although there have been proposals on ways to improve data fidelity [11], quality control remains difficult and sometimes impossible.

It is fair to say that administrative registries can be valuable tools to conduct clinical research, but only as long as their major caveats and limitations are carefully taken into consideration (Table 1). Not only investigators should be careful, but also those reading these studies. Strongly significant results with narrow confidence intervals are not necessarily precise when derived from analyses of administrative databases [12]. The combination of very large cohorts and insufficient adjustment for missing covariates and possible confounders can result in seemingly strong and precise associations that may in fact be weaker or even erroneous. Therefore, a measured interpretation of the results is always advisable.

Table 1 Main advantages and disadvantages of administrative databases when applied to clinical research
Full size table