External validation of nomogram for predicting malignant intraductal papillary mucinous neoplasm (IPMN): from the theory to the clinical practice using the decision curve analysis model

Abstract

The management of IPMNs is a challenging and controversial issue because the risk of malignancy is difficult to predict. The present study aimed to assess the clinical usefulness of two preoperative nomograms for predicting malignancy of IPMNs allowing their proper management. Retrospective study of patients affected by IPMNs. Two nomograms, regarding main (MD) and branch duct (BD) IPMN, respectively, were evaluated. Only patients who underwent pancreatic resection were collected to test the nomograms because a pathological diagnosis was available. The analysis included: 1-logistic regression analysis to calibrate the nomograms; 2-decision curve analysis (DCA) to test the nomograms concerning their clinical usefulness. 98 patients underwent pancreatic resection. The logistic regression showed that, increasing the score of both the MD-IPMN and BD-IPMN nomograms, significantly increases the probability of IPMN high grade or invasive carcinoma (P = 0.029 and P = 0.033, respectively). DCA of MD-IPMN nomogram showed that there were no net benefits with respect to surgical resection in all cases. DCA of BD-IPMN nomogram, showed a net benefit only for threshold probability between 40 and 60%. For these values, useless pancreatic resection should be avoided in 14.8%. The two nomograms allowed a reliable assessment of the malignancy rate. Their clinical usefulness is limited to BD-IPMN with threshold probability of malignancy of 40–60%, in which the patients can be selected better than the “treat all” strategy.

Introduction

Intraductal papillary mucinous neoplasms (IPMNs) may exhibit a spectrum of neoplastic transformation ranging from low-grade dysplasia to high-grade dysplasia (HGD) until invasive carcinomas. Thus, IPMNs have a potential for malignancy, following the “adenoma-carcinoma” sequence, particularly the main duct and mixed forms (50–75%), and to a lesser extent, the BD forms (10–15%) [1,2,3,4]. The management of IPMNs is a challenging and controversial issue. The major effort of the physicians was to perform pancreatic resection mainly for malignant IPMNs because pancreatic surgery inherent morbidity and mortality are not negligible [1, 5]. Several guidelines and consensus conferences 6,7,8,9] stated the indication of surgery, but the percentage of patients who underwent useless pancreatic resection for non-malignant IPMN remains considerable [10, 11]. Therefore, decision-making treatment is often uncertain. Many authors proposed methodologies based on statistics and probability to identify the patients who need surgical resection properly. Among these, preoperative nomograms, basing on variables significantly related to malignant IPMNs, were built 12,13,14,15,16]. The present study aimed to validate the clinical usefulness of preoperative nomograms reported by Attiyeh et al. [12]. The methodology used was a statistical and probabilistic tool called “decision curve analysis (DCA).”

Materials and methods

Study design, patient selection, and nomogram

This is a retrospective study based on a prospectively maintained database of 457 Intraductal Papillary Mucinous Neoplasms (IPMNs) observed from January 2004 to January 2020. The study was approved by the Ethical Committee of S. Orsola-Malpighi Hospital (64/2017/U/Oss) with patient informed consent. The IPMNs types I–II–III were defined according to the consensus conference of Fukuoka 2012 [17]. The diagnostic work-up included Ca 19-9 serum value, Magnetic Resonance Cholangio-Pancreatography (MRCP), and, in selected cases, a multidetector computed tomography (MDCT) and endoscopic ultrasonography (EUS) with or without fine needle biopsy (FNB), were performed. Pancreatic resection was always performed in patients affected by IPMNs with high-risk stigmata according to the consensus conference of Fukuoka 2016 [6] and in selected young patients (< 65 years) with worrisome features. All the other patients underwent surveillance. Only patients who underwent pancreatic resection, both up-front and after a period of follow-up, were included in the analysis because a pathological diagnosis was available. Two nomograms, regarding main and branch duct IPMN, respectively, were evaluated [12] (Figs. 1, 2). For each patient, the data included in the two nomograms were collected: gender, age, symptoms (jaundice and weight loss), tumor site, radiological diagnosis (IPMN types I-III versus type II), solid component/mural enhancing nodules, Wirsung duct size, tumor size, and definitive pathological diagnosis. Also, the type of pancreatic resection and the postoperative data (mortality and morbidity, pancreatic fistula) were reported but not included in the analysis.

Fig. 1
figure1

Clinical nomogram for predicting malignancy in patients with MD-IPMN

Fig. 2
figure2

Clinical nomogram for predicting malignancy in patients with BD-IPMN

Terminology and definition

Postoperative mortality was defined as the number of deaths occurring during hospitalization or within 90 days after surgery. Postoperative morbidity included all complications following surgery up to the day of discharge, according to the Clavien–Dindo classification [18]. A postoperative pancreatic fistula (POPF) was defined according to the 2016 definition proposed by the International Study Group of Pancreatic Fistula (ISGPF) [19].

Statistical analysis and description of decision curve analysis

All the categorical variables were described as frequencies and percentages, while the continuous variables were reported as medians and interquartile ranges. The analysis was performed in two steps. First, a calibration of the score was obtained for both the nomograms, calculating the ability of the score in predicting the probability of malignancy of an IPMN. For this purpose, a logistic regression between the score and malignancy rate was carried out. In relation to the small sample, with the aim to reduce the dispersion risk of the curve, an evaluation of the standard error through the technique of “sandwich estimator of variance,” was performed. Moreover, the nomogram score was simplified in the interval of 20 points. The results were reported for each score point as the post-estimation probability of malignancy within a 95% confidence interval (95% CI). A two-sided P value < 0.05 indicates, for each point, a significant increase in the probability concerning the previous value. Second, both nomograms were tested concerning their clinical usefulness using the decision curve analysis (DCA) 20,21,22,23]. Briefly, decision curve analysis (DCA) is a simple statistical method that allows calculating a clinical benefit for one or more predictions models in comparison to default strategies of treating all or no patients. The DCA includes on the y-axis the “net benefit” and on the x-axis the “threshold probability” (Pt). The net benefit of the model is that it correctly identifies which patients performed a pancreatic resection for IPMN high-grade or invasive carcinoma. Threshold probability refers to how the doctor values the threshold probability of IPMN high-grade or invasive carcinoma for each patient that justifies performing a pancreatic resection. We can assume that the threshold probability (Pt) of a disease at which a patient would opt for treatment is informative of how the patient weighs the relative harms of a false-positive and a false-negative prediction. Thus, the net benefit was calculated as follows:

$$\mathrm{Net}={\left(\frac{\mathrm{TP}}{n}-\frac{\mathrm{FP}}{n}\right)\times \left(\frac{{P}_{\mathrm{t}}}{1-{P}_{\mathrm{t}}}\right)}.$$

In this formula, TP and the FP are the numbers of patients with true- and false-positive results, n is the total number of patients, and Pt is the threshold probability of the disease. This theoretical relationship is then used to derive the net benefit of the model across different threshold probabilities. Plotting net benefit against threshold probability yields the “decision curve.” It was tested for three competing strategies: (1) “to treat all” patients with a pancreatic resection, (2) to “treat none” (3) to select the patients for the pancreatic resection using a nomogram. We also tested if some single factor included in the nomograms predominate over the others. The best model will have the highest Net benefit. We also calculated the useless pancreatic resection avoided for each strategy.

Results

Four-hundred-and-fifty-seven patients affected by IPMN were observed from January 2004 to January 2020. Of these, 98 patients underwent pancreatic resection with pathological diagnosis and were analyzed. The remaining 357 patients were surveilled and were not analyzed. The characteristics of the patients, type of pancreatic resection, and postoperative results are reported in Tables 1 and 2, respectively. The patients were usually female (52.1%), with a median age of 69.7 years (63.6–74.9). Symptoms were not frequent (38.8%), while jaundice and weight loss were sporadic (8.2 and 9.2%, respectively). IPMN was type II in 57.1% of cases, mainly located in the pancreatic head (32.6%), or diffused to the whole pancreas (39.8%). Mural enhancing nodules were present in 57.1%, and the median main duct size was 5 mm, cyst size was ≤ 30 mm in 61.2%. Pathological diagnosis was mainly IPMN high grade and invasive carcinoma (69.4%): malignancy of MD-IPMN resulted in 79.2% of cases, BD-IPMN in 58.5%. The most frequent pancreatic resection performed was distal pancreatectomy (40.8%), severe complications were detected in 14.3%, postoperative mortality in 4.1%. The incidence of clinically relevant pancreatic fistula (grade B and C) was 15.3%. The logistic regression showed that increasing the score of the MD-IPMN nomogram significantly increases the probability of IPMN high grade or invasive carcinoma (beta coefficient = 0.0017 ± 0.008; P = 0.029). The calibration of the MD-IPMN nomogram was reported in Table 3 and plotted in Fig. 3. Each interval of 20 points was significantly related to an increased probability of IPMN high grade or invasive carcinoma. The malignancy rate predicted probability ranges from 48.7% (score = 0 points) to 99.2% (score = 260–280 points). Even if each interval of 20 points is statistically related to the probability of IPMN high grade or invasive carcinoma, starting from score > 140 points, the probability of IPMN high grade or invasive carcinoma increased minimally from 94% (score = 140–159 points) to 99% (score = 260–280 points). The calibration of BD-IPMN nomogram was reported in Table 4 and plotted in Fig. 4. The probability of IPMN high grade or invasive carcinoma was significantly related with the increase of the score (beta coefficient = 0.0016 ± 0.008; P = 0.033). The malignancy rate predicted probability ranges from 26% (score = 20–39 points) to 95% (score = 260–280 points). The major increase in the malignancy rate was obtained from 60 to 120 points (from 41 to 66%).

Table 1 Baseline characteristics of 98 patients affected by IPMNs included in the analysis
Table 2 Post-operative results of 98 operated patients affected by IPMN
Table 3 Calibration of MD-IPMN nomogram score
Fig. 3
figure3

Calibration curve of MD-IPMN nomogram

Table 4 Calibration of BD-IPMN nomogram score
Fig. 4
figure4

Calibration curve of BD-IPMN nomogram

The usefulness of both nomograms was reported in two DCA curves (Figs. 5, 6) for MD-IPMN and BD-IPMN nomograms, respectively. The net benefits and the number of useless pancreatic resection avoided were reported in Tables 5 and 6.

Fig. 5
figure5

Decision curve analysis of MD-IPMN includes three main strategies: to treat all patients; to treat no patients; to treat the patients using nomogram as instrument of selection. The parameters of nomogram were reported also as single factor. Net benefit represents the patients correctly treated. The threshold probability represents the odd of malignancy for which the physician considered acceptable the surgical risk. The use of nomogram does not provide any advantage for any threshold probability of malignancy

Fig. 6
figure6

Decision curve analysis of BD-IPMN includes three main strategies: to treat all patients; to treat non patients; to treat the patients using nomogram as instrument of selection. The parameters of nomogram were reported also as single factor. Net benefit represents the patients correctly treated. The threshold probability represents the odd of malignancy for which the physician considered acceptable the surgical risk. The use of nomogram provide some advantage for a range of value 40–60% threshold probability of malignancy

Table 5 Net benefit values related to the three approaches in MD-IPMN: “treat all,” “treat none,” and “treat use the nomogram”
Table 6 Net benefit values related to the three approaches in BD-IPMN: “treat all,” “treat none,” and “treat use the nomogram”

About MD-IPMN nomogram, Fig. 5 suggested that net values related to the use of nomogram are never superior to those obtained performing the surgical resection in all cases. The net benefit “nomogram” ranged from 79.2 to 27.3%, starting from a threshold probability of 1% until 70%. Net benefit “treat all,” and net benefit “nomogram” resulted similar for the different value of threshold probability of malignancy until the value of 50%. For threshold values of 60% and 70%, the net benefit “treat all” was better than the net benefit “nomogram” (47.2 and 29.1% versus 45.9 and 27.3%, respectively). In addition, useless pancreatic resection avoided resulted 0% and, for value of 60 and 70%, it was negative (− 81.99 and − 66.2%). About BD-IPMN nomogram, Fig. 6 suggested that the use of nomogram produces the highest net benefits only for threshold probability between 40 and 60% (incremental net benefit nomogram = 23.2%). For these values, a maximum of 14.8% of useless pancreatic resection should be avoided. For value inferior to 40%, and superior to 60%, the use of nomogram did not represent the best choice. For threshold value > 70%, the net benefit “nomogram” decreased to 4.7%, − 8.1% and 0%.

Discussion

Although the 2016 Consensus conference of Fukuoka [6] clearly stated when pancreatic resection is recommended for MD-IPMN and BD-IPMN, the optimal treatment remains controversial. Indeed, a large percentage of patients affected by both MD-IPMN and BD-IPMN who underwent pancreatic resection did not present a malignant IPMN. The effort of this study was to validate the use of two nomograms designed to predict the presence of high-grade dysplasia/invasive carcinoma in both MD-IPMN and BD-IPMN. The DCA method was used because it seems particularly suitable in this setting in which a risk of a wrong choice could be not negligible. The advantage of this model, in contrast, to the standard measures, such as the accuracy, was that the area under the curve (AUC) metric focused solely on the predictive accuracy of a model. In other words, in contrast to AUCs, DCA suggests whether the model is worth using at all or which of other more models is preferable [24]. The present study showed that the two nomograms were statistically well-calibrated because the logistic regressions assessed for both nomograms have a significant ability in predicting the presence of high-grade IPMN or invasive carcinoma, increasing the values of the score. This datum means that the malignancy rate predicted is reliable, and it seems to represent a useful parameter for decision-making treatment. In particular, the first model (related to MD-IPMN) showed that starting from score > 140 points, the probability of IPMN high grade or invasive carcinoma increased minimally. In other words, the prevalence of malignant IPMNs resulted very high for score > 140 points, and further distinction appeared useless. Thus, the nomogram for MD-IPMN is useless from value > 140 points. On the other hand, the second model (related to BD-IPMN) showed a slight increase in the malignancy rate with a delayed plateau (Fig. 4). However, from 60 to 120 points, it seems that the malignancy probability increases strongly (from 41 to 66%). This datum means that, in this interval of points, the patients can be selected in the best way.

The DCA allowed different results regarding the clinical usefulness of the two nomograms. Regarding the MD-IPMN nomogram, it is not able to select furtherly the patients with a high risk of malignancy respect to “treat all” strategy. Also, the nomogram is not useful in avoiding useless pancreatic resection. Finally, for the value of threshold probability > 50%, the nomogram resulted less useful than to “treat all” strategy. Indeed, if we consider suitable for surgery all patients having a risk of malignancy at least (threshold probability) of 70%, the treatment strategy based on nomogram will have a net benefit of 27.3% against the 29.1% for a treatment strategy that provides to treat all the cohort of patients affected by MD-IPMN. In summary, the high rate of malignancy (79.2%) of MD-IPMN makes useless an instrument for the selection of patients, such as the nomogram. Henceforth, the optimal strategy is to perform a pancreatic resection in all the patients affected by MD-IPMN, obviously if fit for surgery, as stated by consensus conference of Fukuoka 2016 [6]. Regarding the nomogram related to BD-IPMN, there are some differences in relation to the threshold probability of malignancy: 1-accepting a low threshold probability of malignancy (< 40%), the nomogram allowed the same results of the “treat all” strategy. In other words, if the main goal is to operate all patients even if the risk of malignancy is low, the nomogram is useless to select the patients adequately; 2-on the contrary, if we considered a threshold probability of malignancy between 40 to 60%, the nomogram allowed a net benefit until 23% respect to the strategy “treat all.” Besides, in this range of value, the nomogram allowed to avoid useless pancreatic resection in 14.8% of cases; 3-finally, if we considered only a very high value of the threshold probability of malignancy (> 60%), the nomogram resulted inferior to the strategy proposed by a single parameter (male gender). Thus, if we decide to operate only patients with a threshold probability of malignancy > 60%, the nomogram is not clinically useful, and it is not able to adequately select the patients for the proper strategy treatment. In summary, the nomogram related to BD-IPMN was clinically useful only in the range between 40 and 60% of the threshold probability of malignancy. Henceforth, a “super-selection” that minimizes close to 0, the useless pancreatic resection and, maximize to 100%, the rate of true positive was not possible with this tool.

The present study has several limitations. First, the models were constructed using a small sample size and retrospective data from a prospective single-center database. Second, this is a surgical population and, thus, for definition already super selected. Nonetheless, the use of the DCA approach and the availability of threshold probability reduced the risk due to selection bias typical of the surgical population.

In conclusion, the two nomograms were statistically well-calibrated, allowing a reliable assessment of the malignancy rate (HGD and invasive carcinoma) of both MD-IPMN and BD-IPMN. However, the nomogram related to the MD-IPMN did not result clinically usefulness because it is not able to make a better selection of patients compared with the treatment strategy “treat all”. On the other hand, the nomogram related to BD-IPMN seems to be clinically useful only in a range of value of the threshold probability of malignancy (40–60%) in which it can select the patients better than the “treat all” strategy.

Availability of data and material

The data reported are available and are included in our database of pancreatic neoplasms.

Code availability

n.64/2017/U/Oss.

References

  1. 1.

    Malleo G, Marchegiani G, Borin A et al (2015) Observational study of the incidence of pancreatic and extrapancreatic malignancies during surveillance of patients with branch-duct intraductal papillary mucinous neoplasm. Ann Surg 261:984–990

    Article  Google Scholar 

  2. 2.

    Bournet B, Kirzin S, Carrère N et al (2009) Clinical fate of branch duct and mixed forms of intraductal papillary mucinous neoplasia of the pancreas. J GastroenterolHepatol 24:1211–1217

    Google Scholar 

  3. 3.

    Ricci C, Casadei R, Taffurelli G et al (2016) Risk factors for malignancy of branch-duct Intraductal papillary mucinous neoplasms: a critical evaluation of the Fukuoka guidelines with a systematic review and meta-analysis. Pancreas 45:1243–1254

    Article  Google Scholar 

  4. 4.

    Casadei R, Ricci C, Taffurelli G et al (2018) Impact of surgery and surveillance in the management of branch duct intraductal papillary mucinous neoplasms of the pancreas according to Fukuoka guidelines: the Bologna experience. UpdatSurg 70:47–55

    Google Scholar 

  5. 5.

    Daudé M, Muscari F, Buscail C et al (2015) Outcomes of nonresected main-duct intraductal papillary mucinous neoplasms of the pancreas. World J Gastroenterol 21:2658–2667

    Article  Google Scholar 

  6. 6.

    Tanaka M, Fernández-Del Castillo C, Kamisawa T et al (2017) Revisions of international consensus Fukuoka guidelines for the management of IPMN of the pancreas. Pancreatology 17:738–753

    Article  Google Scholar 

  7. 7.

    European Study Group on Cystic Tumours of the Pancreas (2018) European evidence-based guidelines on pancreatic cystic neoplasms. Gut 67:789–804

    Article  Google Scholar 

  8. 8.

    Vege SS, Ziring B, Jain R et al (2015) American gastroenterological association institute guideline on the diagnosis and management of asymptomatic neoplastic pancreatic cysts. Gastroenterology 148:819–822

    Article  Google Scholar 

  9. 9.

    Italian Association of Hospital Gastroenterologists and Endoscopists, Italian Association for the Study of the Pancreas, Buscarini E, Pezzilli R, Cannizzaro R et al (2014) Italian consensus guidelines for the diagnostic work-up and follow-up of cystic pancreatic neoplasms. Dig. Liver Dis 46:479–493

    Article  Google Scholar 

  10. 10.

    Waters JA, Schmidt CM, Pinchot JW et al (2008) CT vs MRCP: optimal classification of IPMN type and extent. J GastrointestSurg 12:101–109

    Article  Google Scholar 

  11. 11.

    Baiocchi GL, Portolani N, Missale G et al (2010) Intraductal papillary mucinous neoplasm of the pancreas (IPMN): clinicopathological correlations and surgical indications. World J SurgOncol 8:25–31

    Google Scholar 

  12. 12.

    Attiyeh MA, Fernández-Del Castillo C, Al Efishat M et al (2018) Development and validation of a multi-institutional preoperative nomogram for predicting grade of dysplasia in intraductal papillary mucinous neoplasms (IPMNs) of the pancreas. Ann Surg 267:157–163

    Article  Google Scholar 

  13. 13.

    Correa-Gallego C, Do R, Lafemina J, Gonen M et al (2013) Predicting dysplasia and invasive carcinoma in intraductal papillary mucinous neoplasms of the pancreas: development of a preoperative nomogram. Ann SurgOncol 20:4348–4355

    Google Scholar 

  14. 14.

    Hijioka S, Shimizu Y, Mizuno N et al (2014) Can long-term follow-up strategies be determined using a nomogram-based prediction model of malignancy among intraductal papillary mucinous neoplasms of the pancreas? Pancreas 43:367–372

    CAS  Article  Google Scholar 

  15. 15.

    Shimizu Y, Yamaue H, Maguchi H et al (2015) Validation of a nomogram for predicting the probability of carcinoma in patients with intraductal papillary mucinous neoplasm in 180 pancreatic resection patients at 3 high-volume centers. Pancreas 44:459–464

    PubMed  Google Scholar 

  16. 16.

    Jang JY, Park T, Lee S et al (2017) Proposed nomogram predicting the individual risk of malignancy in the patients with branch duct type intraductal papillary mucinous neoplasms of the pancreas. Ann Surg 266:1062–1068

    Article  Google Scholar 

  17. 17

    Tanaka M, Fernández-del Castillo C, Adsay V et al (2012) International Association of Pancreatology. International consensus guidelines 2012 for the management of IPMN and MCN of the pancreas. Pancreatology 12:183–197

    Article  Google Scholar 

  18. 18.

    Casadei R, Ricci C, Pezzilli R et al (2011) Assessment of complications according to the Clavien–Dindo classification after distal pancreatectomy. JOP 12:126–130

    PubMed  Google Scholar 

  19. 19.

    Bassi C, Marchegiani G, Dervenis C et al (2017) The 2016 update of the International Study Group (ISGPS) definition and grading of postoperative pancreatic fistula: 11 years after. Surgery 161:584–591

    Article  Google Scholar 

  20. 20.

    Pauker SG, Kassirer JP (1980) The threshold approach to clinical decision making. NEJM 302:1109–1117

    CAS  Article  Google Scholar 

  21. 21.

    Tsalatsanis A, Hozo I, Vickers A et al (2010) Regret theory approach to decision curve analysis: a novel method for eliciting decision makers’ preferences and decision-making. BMC Med Inform DecisMak 10:51–55

    Article  Google Scholar 

  22. 22.

    Vickers AJ, Elkin EB (2006) Decision curve analysis: a novel method for evaluating prediction models. Med DecisMak 26:565–574

    Google Scholar 

  23. 23.

    Vickers AJ, van Calster B, Steyerberg EW (2019) A simple, step-by-step guide to interpreting decision curve analysis. DiagnProgn Res 4:3–18

    Google Scholar 

  24. 24.

    Djulbegovic B, Desoky AH (1996) Equation and nomogram for calculation of testing and treatment thresholds. Med DecisMak 16:198–199

    CAS  Google Scholar 

Download references

Funding

Open access funding provided by Alma Mater Studiorum - Università di Bologna within the CRUI-CARE Agreement.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Riccardo Casadei.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Research involving human participants and/or animals

The study was conducted according to the principles of the Declaration of Helsinki (64th version, October 2013) and in accordance with the local laws and regulations.

Informed consent

All participants provided informed consent prior to thei participation.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Casadei, R., Ricci, C., Ingaldi, C. et al. External validation of nomogram for predicting malignant intraductal papillary mucinous neoplasm (IPMN): from the theory to the clinical practice using the decision curve analysis model. Updates Surg (2021). https://doi.org/10.1007/s13304-021-00999-4

Download citation

Keywords

  • Pancreas
  • Intraductal papillary mucinous neoplasms
  • Nomogram
  • Dysplasia