Abstract
Purpose
A German multicentre study BLOOMY was the first to use machine learning approach to develop mortality prediction scores for bloodstream infection (BSI) patients, but the scores have not been assessed in other cohorts. Our aim was to assess how the BLOOMY 14-day and 6-month scores estimate mortality in our cohort of 497 cases with BSI.
Methods
Clinical data, laboratory data, and patient outcome were gathered retrospectively from patient records. The scores were calculated as presented in the BLOOMY study with the exception in the day of the evaluation.
Results
In our cohort, BLOOMY 14-day score estimated death by day 14 with an area under curve (AUC) of 0.87 (95% Confidence Interval 0.80–0.94). Using ≥ 6 points as a cutoff, sensitivity was 68.8%, specificity 88.1%, positive predictive value (PPV) 39.3%, and negative predictive value (NPV) 96.2%. These results were similar in the original BLOOMY cohort and outweighed both quick Sepsis-Related Organ Failure Assessment (AUC 0.76) and Pitt Bacteraemia Score (AUC 0.79) in our cohort. BLOOMY 6-month score to estimate 6-month mortality had an AUC of 0.79 (0.73–0.85). Using ≥ 6 points as a cutoff, sensitivity was 98.3%, specificity 10.7%, PPV 25.7%, and NPV 95.2%. AUCs of 6-month score to estimate 1-year and 5-year mortality were 0.80 (0.74–0.85) and 0.77 (0.73–0.82), respectively.
Conclusion
The BLOOMY 14-day and 6-month scores performed well in the estimations of mortality in our cohort and exceeded some established scores, but their adoption in clinical work remains to be seen.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Various scores have been developed for emergency department and hospitalised patients to determine the degree of illness or to predict an endpoint such as mortality. An example of a widely used score for all patients is National Early Warning Score 2 [1]. The vital signs or other parameters needed for the score are inserted in an electronic patient record and the record automatically shows you, for example, the trend of the score. Another example is Sequential Organ Failure Assessment (SOFA) score [2], commonly used for all intensive care unit (ICU) patients to predict mortality. The newcomers, BLOOMY 14-day and 6-month prediction scores [3], are developed for bloodstream infection (BSI) patients, they need to be calculated electronically, and they have not been yet studied in other cohorts than in the original BLOOMY study. Before roll-out, more studies are needed to test their reproducibility and validity [4,5,6,7].
The BLOOMY study is a multicentre cohort study at six German tertiary care university hospitals. It was the first study to develop a prediction score for patients with infection by use of machine-learning approach. The 14-day and 6-month mortality model predictors included, for example, age, body-mass index, platelet counts and C-reactive protein concentrations (see Tables 1 and 2). The total possible scores for the BLOOMY 14-day and 6-month score were 20 and 26, respectively. In prediction of mortality, sensitivity increased with higher scores but specificity decreased; the study group used mostly 6 or 7 points as cutoff. The researchers had a separate validation cohort to calculate predictive accuracy. The aim of our study was to assess how the score estimates mortality in our cohort of 497 Finnish BSI patients, i.e., to externally assess the sensitivity, specificity, and other predictive values. We also studied how quick SOFA (qSOFA) is comparable to BLOOMY 14-day score and tested how BLOOMY 6-month score predicts 1-year and 5-year mortality.
Materials and methods
Bacteraemia patient cohort
Tampere University Hospital is a tertiary hospital with a catchment population of ca. 525 000 inhabitants in the Pirkanmaa County, Finland. The study cohort included consecutive blood culture-positive patients in the Tampere University Hospital ED between March 1, 2012 and February 28, 2014. The cases in which positive blood culture were determined as contamination were excluded. The clinical data of the patients were collected retrospectively from patient records. A criterion for the qSOFA score was calculated based on Sepsis‐3 definitions [8]. The Pitt Bacteraemia Score was calculated as presented by Korvick et al. [9]. The site of infection was determined retrospectively. A more detailed description of the patient cohort is available in our previous publications [10,11,12]. The study was approved by the Ethics Committee of Tampere University Hospital, Finland (permit# R11099). The need for informed consent was waived as routine patient care was not modified.
Variables
The BLOOMY 14-day and 6-month scores were calculated as described in the original article apart from two modifications due to practical reasons. On BLOOMY 14-day scores, day 0 variables were used instead of day 3 variables. This has an effect on points allocated to hypotension, mental status, and laboratory values (see Table 1), but not on the permanent patient data such as malignancies and medication. BLOOMY 6-month score uses laboratory variables minimum of six days after the last day of antibiotic treatment (Table 2). Instead, day 0 laboratory parameters were used. Again, permanent patient date was collected as in original article. In dichotomous variables, only cases with missing data that would affect the result were excluded. In continuous variables all cases with missing data were excluded. Patients with multiple admissions (n = 12) were handled as they would be separate cases.
Statistical analyses
SPSS version 28.0 software (IBM Corp., NY, USA) was used for statistical analyses. AUC analysis was used to assess the scores’ discriminative value and specificity, sensitivity, negative predictive value (NPV), and positive predictive value (PPV) was used to assess the predictive accuracy. A P‐value of < 0.05 was considered significant. To assess the calibration of the models, the calibration belt was used [7, 13].
Results
A total of 497 cases (484 patients) with positive blood culture were included. Median age was 68 (range 16–95 years), 53% were male, and Gram-positive and Gram-negative bacteraemia were equally common. Other characteristics, the severity of the disease, and causative organism are shown in Table 3. Ninety-nine cases were missing weight and 3 cases day of admission C-reactive protein, no other missing data for the BLOOMY score calculations. We found no need to recalibrate the scores (see supplement report).
We calculated the predictive value of BLOOMY 14-day score to estimate death by day 14. AUC was 0.87 (95% Confidence Interval 0.80–0.94). Using ≥ 6 points as a cutoff, sensitivity was 68.8% and specificity 88.1%. With ≥ 7 points, sensitivity was 42.9%, specificity 94.9%, PPV 48.8%, and NPV 93.5%. In our cohort, Pitt Bacteraemia Score had an AUC of 0.79 (0.72–0.86) to estimate 14-day mortality. For the same estimation, ≥ 2 qSOFA points had an AUC of 0.76 (0.69–0.83).
Using ≥ 6 points as a cutoff, BLOOMY 6-month score estimated 6-month mortality with a sensitivity of 98.3% and a specificity of 10.7%. Increasing cutoff to ≥ 7 points decreased sensitivity to 93.2% and increased specificity to 21.0% (data on 483 cases). BLOOMY 6-month score had an AUC of 0.79 (0.73–0.85). AUCs of 6-month score to estimate 1-year and 5-year mortality were 0.80 (0.74–0.85) and 0.77 (0.73–0.82), respectively; see Table 4 for other predictive values for both BLOOMY 14-day and 6-month score.
The quick BLOOMY 14-day score included three parameters: age, mental status, and hypotension. The predictive value (AUC) of quick BLOOMY 14-day score to estimate 14-day mortality was 0.81 (0.75–0.87). Quick BLOOMY score of ≥ 6 points predicted 19 of 55 deaths at 14 days with a sensitivity of 34.6%, a specificity of 95.5%, a PPV of 48.7%, and an NPV of 92.1%.
Discussion
We calculated the predictive value of BLOOMY 14-day score to estimate death by day 14. In our study AUC was 0.87, almost the same as in the original BLOOMY cohort (0.86). Using 6 points or more as a cutoff, our sensitivity to predict day 14 mortality was 68.8% (original 61.3%) and specificity 88.1% (original 86.4%). Quick BLOOMY 14-day score AUC was 0.81 in our cohort vs. 0·83 in the BLOOMY cohort. The AUC of Pitt Bacteraemia Score to estimate 14-day mortality was 0.79, almost identical than in the original cohort (0.78). The AUC of qSOFA was 0.76 in our study. In the BLOOMY study qSOFA was not calculated.
The predictive values of BLOOMY 6-month score to estimate 6-month mortality in our cohort were different from the original BLOOMY cohort. We calculated the score by using day 0 laboratory values instead of, as in the BLOOMY cohort, the last day of antibiotic treatment plus minimum of six days. This affected the specificity of the 6-point cutoff; it was much lower in our cohort (10.7%) than in the BLOOMY cohort (52.8%). However, AUC was still higher than in the original cohort, 0.79 vs. 0.74, respectively.
The BLOOMY study compared the new scores to Pitt Bacteraemia Score, Gram-Negative Bloodstream Infection Risk Mortality Score [14], and, indirectly, to results from a qSOFA meta-analysis [15]. They found the predictive values of BLOOMY 14-day and 6-month scores to be better than in these older scores. Combined with our results, it appears that the machine learning approach has developed a score that yields better 14-day and 6-month mortality estimations than these established scores.
BLOOMY 14-day and 6-month prediction scores are developed for BSI patients, and they need to be calculated electronically. One prediction score developed for patients with infection, e.g. qSOFA, is, in contrast to all BLOOMY scores, easily calculated bedside and therefore gained popularity. According to our experience, other prediction scores developed only for patients with infection have not been widely used in clinical settings. Pitt Bacteraemia Score would need either electronic record-based, web-based, or app-based calculation to obtain the results, but these calculations are rarely performed. The same might happen with these newcomers.
Our study had some limitations. We were able to calculate all BLOOMY scores retrospectively, but some values were missing. Ninety-nine cases were missing weight which is used in BLOOMY 14-day and 6-month score (BMI ≤ 20 kg/m2 equals 1 point). We decided not to reduce the total number to those with weight as the missing values affected less on the dichotomous variables (see Table 4). We had > 99% of other values. Other limitation is the difference in the days of evaluation in the original cohort and in our cohort. The original cohort used day 3 as an evaluation point in BLOOMY 14-day score and the last day of antibiotic treatment plus minimum of 6 days in BLOOMY 6-month score. In the present study, day 0 was used as an evaluation point in both BLOOMY 14-day and in the laboratory results of BLOOMY 6-month score. The difference in BLOOMY 14-day evaluation day can also been seen as an advantage of the present study, but the difference in BLOOMY 6-month might have been the reason for the differences in the sensitivity and specificity in the original and this study as the laboratory results add up to maximum of 10 points out of 26 total points. On the other hand, in clinical work, laboratory tests are often not routinely taken 6 days after antibiotic treatment.
In conclusion, the BLOOMY 14-day and BLOOMY 6-month scores performed well in our cohort. However, it is more likely that any new score is adopted in clinical routine if it is for general patient population and not a score for cases with a certain laboratory result (i.e. positive blood culture). It is possible that these new scores will remain mostly in research purposes.
Data availability
Data available on request.
References
Royal College of Physicians. National Early Warning Score (NEWS) 2 [Internet]. https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2
Vincent JL, Moreno R, Takala J, Willatts S, De Mendonça A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22:707–10.
Tacconelli E, Göpel S, Gladstone BP, Eisenbeis S, Hölzl F, Buhl M, et al. Development and validation of BLOOMY prediction scores for 14-day and 6-month mortality in hospitalised adults with bloodstream infections: a multicentre, prospective, cohort study. Lancet Infect Dis. 2022;22:731–41.
Adegbite BR, Edoa JR, Ndzebe Ndoumba WF, Dimessa Mbadinga LB, Mombo-Ngoma G, Jacob ST, et al. A comparison of different scores for diagnosis and mortality prediction of adults with sepsis in low-and-middle-income countries: a systematic review and meta-analysis. eClinicalMedicine. 2021;42: 101184.
Juthani-Mehta M, Quagliarello VJ. Prognostic scoring systems for infectious diseases: their applicability to the care of older adults. Clin Infect Dis. 2004;38:692–6.
Justice AC. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130:515.
Schuetz P, Koller M, Christ-Crain M, Steyerberg E, Stolz D, Müller C, et al. Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings. Epidemiol Infect. 2008;136:1628–37.
Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801–10.
Korvick JA, Marsh JW, Starzl TE, Yu VL. Pseudomonas aeruginosa bacteremia in patients undergoing liver transplantation: an emerging problem. Surgery. 1991;109:62–8.
Rannikko J, Syrjänen J, Seiskari T, Aittoniemi J, Huttunen R. Sepsis-related mortality in 497 cases with blood culture-positive sepsis in an emergency department. Int J Infect Dis. 2017;58:52–7.
Rannikko J, Seiskari T, Huttunen R, Tarkiainen I, Jylhävä J, Hurme M, et al. Plasma cell-free DNA and qSOFA score predict 7-day mortality in 481 emergency department bacteraemia patients. J Intern Med. 2018;284:418–26.
Rannikko J, Jacome Sanz D, Ortutay Z, Seiskari T, Aittoniemi J, Huttunen R, et al. Reduced plasma PCSK9 response in patients with bacteraemia is associated with mortality. J Intern Med. 2019;286:553–61.
Nattino G, Lemeshow S, Phillips G, Finazzi S, Bertolini G. Assessing the calibration of dichotomous outcome models with the calibration belt. Stata J. 2017;17:1003–14.
Al-Hasan MN, Lahr BD, Eckel-Passow JE, Baddour LM. Predictive scoring model of mortality in Gram-negative bloodstream infection. Clin Microbiol Infect. 2013;19:948–54.
Song J-U, Sin CK, Park HK, Shim SR, Lee J. Performance of the quick Sequential (sepsis-related) Organ Failure Assessment score as a prognostic tool in infected patients outside the intensive care unit: a systematic review and meta-analysis. Crit Care. 2018;22:28.
Acknowledgements
None.
Funding
Open access funding provided by Tampere University (including Tampere University Hospital). This work was supported by the Competitive State Research Financing of the Expert Responsibility Area of Tampere University Hospital (Grant 9N075 to J.S. and X50060 to J.A) The authors’ work was independent of the funder (the funding source had no involvement).
Author information
Authors and Affiliations
Contributions
JT: conceptualization, data curation, formal analysis, investigation, methodology, project administration, software, writing—original draft; TS: conceptualization, investigation, methodology, writing—review and editing; JA: conceptualization, formal analysis, investigation, methodology; HH: methodology, software, validation; JM: funding acquisition, resources, writing—review and editing; RH: conceptualization, funding acquisition, project administration, supervision, writing—review and editing; JS: funding acquisition, project administration, supervision, writing—review and editing; JR: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, project administration, supervision, writing—original draft.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Tietäväinen, J., Seiskari, T., Aittoniemi, J. et al. Assessment of a novel BLOOMY score for predicting mortality in hospitalised adults with bloodstream infection. Infection 52, 1511–1517 (2024). https://doi.org/10.1007/s15010-024-02254-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s15010-024-02254-5