Using text analysis software to identify determinants of inappropriate clinical question reporting and diagnostic procedure referrals in Reggio Emilia, Italy



Inappropriate prescribing of diagnostic procedures leads to overdiagnosis, overtreatment and resource waste in healthcare systems. Effective strategies to measure and to overcome inappropriateness are essential to increasing the value and sustainability of care.

We aimed to describe the determinants of inappropriate reporting of the clinical question and of inappropriate imaging and endoscopy referrals through an analysis of general practitioners’ (GP) referral forms in the province of Reggio Emilia, Italy.


A clinical audit was conducted on routinely collected referral forms of all GPs of Reggio Emilia province. All prescriptions for gastroscopy, colonoscopy, neurological and musculoskeletal computerised tomography (CT) and magnetic resonance imaging (MRI) from 2012 to 2017 were included. The appropriateness of referral forms was assessed using Clinika VAP software, which combines semantic analysis of clinical questions and available metadata. Local protocols agreed on by all physicians defined criteria of appropriateness. Two multilevel logistic models were used to identify multiple predictors of inappropriateness of referral forms and to analyse variability among GPs, primary care subdistricts and healthcare districts.


Overall, 37% of referral forms were classified as inappropriate, gastroscopy and CT showed higher proportions of inappropriate referrals compared to colonoscopy and MRI. Inappropriateness increased with patient age for CT and MRI; for gastroscopy, it was lower for patients aged 65–84 compared to those younger, and for colonoscopy, it was higher for older patients. Fee exemptions were associated with inappropriateness in MRI referral forms. The effect of GPs’ practice organization was consistent across all tests, showing higher inappropriateness for primary care medical networks than in primary care medical groups. Male GPs were associated with inappropriateness in endoscopy, and older GPs were associated with inappropriateness in musculoskeletal CT. While there was moderate variability in the inappropriate prescribing among GPs, there was not among the healthcare districts or primary care subdistricts.


Routinely collected data and IT tools can be useful to identify and monitor diagnostic procedures at high risk of inappropriate prescribing. Assessing determinants of inappropriate referral makes it possible to tailor educational and organizational interventions to those who need them.

Peer Review reports


The burden of inappropriate use of diagnostic procedures can be seen worldwide; its impact on patients’ health and quality of life can include overdiagnosis, overexposure to radiation, complication due to invasive procedures and overtreatment [1,2,3]. Moreover, its impact on sustainability is an additional issue, especially in universal health care systems, where it leads to resource waste, increase in costs and longer waiting lists [4].

Thus, tackling inappropriateness is worthwhile. To do so requires identifying its drivers; strategies to overcome barriers to appropriate prescribing would thus increase the value of care [3]. In 2017, experts from 19 countries who are members of the European Society of Radiology defined a list of critical issues and needs to increase appropriateness in the use of diagnostic imaging procedures [5], including the need for evidence-based imaging referral guidelines [5,6,7], the need to justify diagnostic procedures, particularly invasive procedures using ionizing radiation or contrast medium, and the need for general practitioners and specialists to work jointly [5, 8]. Another issue is that of defensive medicine, which can lead to an increase in diagnostic procedure prescriptions, although this appears to only partially explain the whole phenomenon [9]. Inappropriate prescription of gastrointestinal endoscopic procedures is also considered critical primarily for two reasons: they are invasive procedures, which cause considerable discomfort for the patient, and they have an intrinsic, although low, risk of serious complications, particularly colonoscopy [10]. Endoscopy is the main bottleneck in colorectal screening programmes in most countries; it surely is in Italy [11, 12], having become the main barrier to scaling up one of the most effective and cost-effective prevention interventions [13]. Thus, inappropriate referrals compete for allocation of resources to other, more useful and often more urgent procedures, thereby reducing the allocative efficiency of the system [14].

Although the drivers of and barriers to appropriate use of diagnostic resources are similar globally, actions should be taken at both the national and the local level [5].

In the province of Reggio Emilia (Emilia-Romagna region, northeastern Italy), a multi-component intervention has been in place since 2008 to improve the clinical and organizational appropriateness of diagnostic procedures. A multidisciplinary working group made up of all the Reggio Emilia Local Health Authority healthcare professionals and stakeholders defined a list of criteria for prescribing neurological and musculoskeletal magnetic resonance imaging (MRI) and computed tomography (CT) and for gastroscopy and colonoscopy.

This information was used by a software to define and assess specific types of misreporting and clinical inappropriateness of all referral forms for the diagnostic procedures listed above, routinely entered in an electronic database.


The aim of this study was to describe the determinants of incomplete or meaningless reporting of the clinical question, of inappropriate imaging and endoscopy referrals and of an inappropriate level of urgency indicated in the referral through an analysis of general practitioners’ (GP) referral forms in the province of Reggio Emilia, Italy.


Study design

The analyses here reported are part of a comprehensive clinical audit cycle in a quality improvement system. A trend analysis of the main indicators is reported. Cross-sectional analysis of the association between inappropriateness of the prescription and patients’, GPs’ and healthcare organization’s characteristics for the period 2012–2017 was conducted on routinely collected data.

Setting and population

This study was implemented within the Local Health Authority of Reggio Emilia, which provides healthcare services for the whole population of the province of Reggio Emilia (approximatively 530,000 inhabitants in 2019). The province is divided into 6 healthcare districts, with 23 primary care subdistricts (PCSDs) in which about 308 general practitioners (GP) operate. In each PCSD, the GPs are organised in practices sharing the same clinic and infrastructure and in which the beneficiaries are assisted by any one of the GPs in the group (primary care medical group), or where beneficiaries are assisted by their own GP, who shares some IT infrastructures with other GPs (primary care medical network) (See Additional file 1: Appendix Table 1 for details).

The Italian National Health Service is based on principles of universalism and comprehensiveness, and the Ministry of Health has the exclusive mandate to set the so-called “essential levels of care” (LEA), the list of healthcare services which must be guaranteed to all citizens and resident foreigners. The Italian Regional Healthcare Services allocate and administer the public funding for and establish the organization of providing the LEA. Only the healthcare services indicated in the LEA are available for free. In this context, GPs have the task of filtering access of their patients to specialist outpatient services, per the gatekeeping model. GPs can request various health services through a computerized referral form which should include a diagnostic question and relevant information on the clinical condition of the patient.

Data sources

In Emilia-Romagna Region, all the computerized referral forms of outpatient care are routinely recorded in a specific database; the information collected includes the patient’s personal information, the name of the referring physician, the date of prescription, the clinical question or health condition requiring the procedure and the procedure/test requested.

In this study we analysed all the general practitioners’ referral forms in the period 2012–2017 for colonoscopy, gastroscopy, neuro CT, musculoskeletal CT, neuro MRI and musculoskeletal MRI performed by public and private providers operating for the National Health Service in the province of Reggio Emilia. Approximately 323,000 referral forms were written by GPs of the province, accounting for 85% of all referral forms in the study period.


In the province of Reggio Emilia, the clinical question on referral forms for diagnostic procedures has been mandatory since 2005. Moreover, in 2008, a multidisciplinary working group including all healthcare professionals and stakeholders defined a list of criteria for prescribing neurological and musculoskeletal magnetic resonance imaging (MRI) and computed tomography (CT) and for gastroscopy and colonoscopy on the basis of national and international guidelines. The criteria also defined the level of urgency (i.e. urgent (U), deferred urgency (B), to be scheduled within 60 days (D) or planned follow-up (P)) appropriate for each pair of clinical question/ requested test.

The multidisciplinary working group implemented an educational and retraining programme for GPs and specialists.

Software for the analysis of clinical questions was developed and implemented to assess the appropriateness of referral forms for colonoscopy, gastroscopy, neuro CT, musculoskeletal CT, neuro MRI and musculoskeletal MRI. This assessment was based on criteria included in the developed provincial protocols. The software uses features designed to process “unstructured clinical information” to recognize and make available the embedded knowledge items (i.e. “semantic analysis engine optimized for the ontology and thesaurus of a clinical field, with over 2 million concepts constantly updated”).

The software used for the assessment is called Clinika / VAP, produced by IG Consulting (Maps Group). It can be used for either an ex-ante assessment (i.e. to support the doctor in filling out the referral form) or an ex-post assessment (i.e. to perform periodic systematic assessments on prescribing behaviour).

The text of the clinical questions undergoes a semantic analysis to recognize and organize the clinical concepts indicated by the prescriber. All types of clinical concepts concerning “conditions/diseases” are classified and linked with the appropriate domain language. Subsequently, the prescribing rules are interpolated in the diagnostic procedure required in order to verify that the condition/disease identified in the clinical question is one of those admitted by the protocol and to obtain the priority admitted by the protocol for this type of combination (i.e. diagnostic procedure – condition/disease). The priority requested on the referral form and the priority foreseen by the protocol are then compared.

Benchmarking reports are periodically produced and sent to GPs and coordinators of primary care subdistricts to motivate GPs to improve the appropriateness of their diagnostic procedure referral practices.

The appropriateness of diagnostic procedure referral has also been included among the objectives agreed on by the LHA and the primary care subdistricts for granting GPs incentives.


This process assigns each referral form, according to characteristics assessed by the software, to one of the following categories (Fig. 1):

  • Inappropriate:

    • clinical question missing (i.e. incomplete);

    • clinical question present, but with no clinical meaning (i.e. meaningless);

    • meaningful clinical question not matching any criterion included in the protocol for the diagnostic procedure prescribed (i.e. clinically inappropriate condition/test match);

    • clinically appropriate condition/test match but priority level different from that foreseen for the health problem (i.e. inappropriate level of urgency)

  • Appropriate:

    • Clinical question and priority level consistent with the health problem, as indicated by the protocols for the diagnostic procedure prescribed.

Fig. 1

Flowchart of ex-post assessment of overall inappropriateness of referral forms using the Clinika/VAP software. Data shown are related to all referral forms for all six of the included diagnostic procedures. The percentage values refer to the total number of referral forms (i.e. n = 322,978) prescribed by general practitioners in the province of Reggio Emilia, Italy, between 2012 and 2017

The primary endpoint was overall appropriateness, including all missing, meaningless reporting, clinically inappropriate condition/test match and inappropriate level of urgency. The secondary endpoint was to also evaluate only the clinically inappropriate test/condition match, excluding incomplete and meaningless referrals from the analysis. In this analysis, referrals with an inappropriate level of urgency were included among inappropriate referrals because, while this type of inappropriateness is independent of clinical reasons, it does lead to allocative inefficiency and inappropriate use of resources.

An appropriateness evaluation performed by Clinika software and an expert radiologist or gastroenterologist on a sample of 100 records for endoscopy procedures showed an agreement of 78.8% (K = 0.64) for colonoscopy and 85.9% (K = 0.66) for gastroscopy.

Statistical analysis

Descriptive analyses of referral forms were performed for each type of diagnostic procedure prescribed and according to predefined determinants of inappropriateness.

Time trends were calculated for each prescribed diagnostic procedure between 2012 and 2017.

The determinants of overall appropriateness were calendar year, whether a health fee waiver (exemption) was applied (providing diagnostic procedure for free), age and sex of patient and of physician and GP’s practice organization (primary care medical group or primary care medical network).

Random intercept multilevel models that included GPs, primary care subdistricts and healthcare districts as random effects were made [15]. We report the fixed effect of the models: the adjusted odds ratios (ORs) from the multilevel models that included patient-level factors and GP-level factors to identify determinants of the overall inappropriateness of referral forms. Furthermore, we used a multilevel model adjusted for 1st level variables to analyse the variability in inappropriateness of referral forms between GPs (level 2), between primary care subdistricts (level 3) and between healthcare districts (level 4). We summarized the residual variability and interclass correlation coefficients (ICC), which represents the proportion of the total variance in inappropriateness directly attributable to each level after taking into account the division random effects and fixed effects for all variables.

The variability in appropriateness was then assessed at three levels: GP, primary care subdistrict and healthcare district. The multilevel logistic model was used to assess the influence of individual predictors on inappropriateness of referral forms. Let [P[Yijkl = 1]] be the probability that the lth referral form ordered by the kth general practitioner in the jth primary care subdistrict in the ith healthcare district was inappropriate. The model is written out as follows:

$$ logit\left[P\left[{Y}_{ijkl}=1\right]\right]=\mathit{\log}\left[\frac{P\left[{Y}_{ijkl}=1\right]}{1-P\left[{Y}_{ijkl}=1\right]}\right]=\left({\beta}_{0i}+{\beta}_{0j}+{\beta}_{0k}\right)+\theta {X}_{ijkl} $$

where β0i is the healthcare district-specific random intercept, β0j the primary care subdistrict-specific random intercept, β0k the GP-specific random intercept, Xijkl the vector of individual-level covariates and θ the vector of individual-level coefficients.

We used STATA 13.0SE (Stata Corporation, Texas, TX) software package for the main analysis.


This is a clinical audit on routinely collected administrative data. According to the Italian law, clinical audits do not require the approval of an ethics committee. Data are available upon reasonable request by writing to


Descriptive analysis

Overall, 322,978 referral forms were included in the analysis, of which 203,620 (63.0%) proved to be appropriate according to our assessment criteria. Inappropriateness was mainly driven by clinical inappropriateness (i.e. mismatch between health problem, diagnostic procedure prescribed and level of urgency), accounting for 27.7% (n = 89,310/322,978) of all prescribed referral forms, while inappropriate reporting of clinical question (i.e. presence of a clinical question that defines a clear health problem included among the prescribing criteria) accounted for 9.3% (n = 30,048/322,978) of overall inappropriateness (Fig. 1).

The same pattern was found among referral forms for each diagnostic procedure, with greater clinical inappropriateness for endoscopy and CT than for MRI. Assessing referral forms by diagnostic procedure and year of prescription, we observed an overall decrease in inappropriateness, from 44.2% in 2012 to 32.9% in 2017. In particular, an increasing trend (from 2012 to 2016) for all appropriateness endpoints occurred for all diagnostic procedures, with a substantial plateau between 2016 and 2017. Only neuro CT showed a slight reduction in appropriateness in the last year of the observation period, while musculoskeletal CT appropriateness rose during over same period (Fig. 2).

Fig. 2

Trends in proportion of completeness, meaningful, clinically appropriate condition/test match and overall appropriateness of referral forms prescribed by general practitioners from 2012 to 2017 in the province of Reggio Emilia, Italy, using the Clinika /AVP software by diagnostic procedure

The number of referral forms and the overall proportion of inappropriateness varied according to patient, prescription and prescribing physician’s characteristics (Table 1). The lowest proportion of appropriateness was generally found among referral forms provided to older people (i.e. over age 84) and to female patients, with the exception of gastroscopy prescriptions, for which appropriateness was lower in males and in younger patients. Further, the overall proportion of appropriateness decreased with the increasing of the level of urgency, while a similar proportion was found between referral forms including or not including fee exemptions. Regarding prescribers’ characteristics, the lowest proportion of appropriateness was found among prescriptions provided by male and by older physicians. GPs working in a medical network wrote fewer appropriate prescriptions than did GPs working in a medical group. Differences in the proportions of appropriate referral forms were found among the six health districts of the province.

Table 1 Descriptive data for patients’, physicians’ and prescriptions’ characteristics

Determinants of inappropriateness

Inappropriateness increased with patient age for all imaging procedures, while a lower risk of inappropriateness of gastroscopy was found among patients aged 65–84. Moreover, compared to female patients, male patients showed a lower risk of inappropriateness for colonoscopy, musculoskeletal CT, neuro MRI and musculoskeletal MRI and a higher risk for gastroscopy and neuro CT. Referral forms with fee exemptions resulted in greater inappropriateness for musculoskeletal MRI and neuro MRI. For all diagnostic procedures, inappropriateness of prescription was strongly associated with the GP’s work structure. Regarding physician characteristics, male physicians were positively associated with inappropriate colonoscopy (OR = 1.33; 95% CI = 1.14–1.56) and gastroscopy (OR = 1.18; 95% CI = 1.02–1.36), while older physicians were positively associated with inappropriate musculoskeletal CT (OR = 2.17; 95% CI = 1.30–3.62) (Table 2).

Table 2 Multilevel analysis of overall inappropriateness of referral forms: (A) fixed effects; (B) random effects estimates

For all diagnostic procedures, variability between GPs accounted for the highest percentage of total variability in inappropriateness (from 7.1% for neuro CT to 14.3% for musculoskeletal CT), while variability between primary care subdistricts (from 0.0% for neuro CT to 2.5% for head MRI) and healthcare district (from 0.% for MRI and neuro CT to 0.7% for musculoskeletal CT) represented a smaller portion of the total variability in inappropriateness in the statistical model adjusted for patient’s age, sex, year of prescription and exemption status.

We performed a sensitivity analysis to examine the clinical appropriateness of referral forms, excluding incomplete and meaningless referral forms (See Additional file 1: Appendix Table 2–3). The associations were almost all in the same direction of that observed for overall appropriateness, even if in most cases the associations were weaker and estimates less precise. The few exceptions are a negative association for fee exemption in endoscopy (OR = 0.88; 95% CI = 0.83–0.94 for colonoscopy and OR = 0.90; 95% CI = 0.85–0.96 for gastroscopy) and an association with older patient age for colonoscopy, which disappeared.

Variability between GPs in clinical inappropriateness was slightly smaller than in the main analysis, but its stronger effect remained among the hierarchical levels.


More than one-third of prescribed diagnostic procedures in our study were classified as inappropriate, with gastroscopy, neuro CT and musculoskeletal CT having a higher percentage of inappropriate referrals than colonoscopy, neuro MRI and musculoskeletal MRI. Overall, inappropriateness was mostly attributed to clinical inappropriateness (i.e. inconsistency between test, clinical question and urgency reported in the forms, according to prespecified criteria) and less to the inappropriate reporting of clinical question for referral (i.e. “incomplete” or “meaningless” forms). Despite the fact that the variability between GPs was the greatest source of inappropriateness variation, promising improvement was observed over time for all procedures under study, consistent with the implementation of several measures of training, shared protocol definition and administrative control. Both the appropriateness of requesting a diagnostic test and the clinical relevance of test results to the management of a patient require reliable, effective communication between the referring physician and the specialist [3, 5, 16]; the cornerstone of that communication is the quality and structure of a report, which includes all findings and the specialist’s interpretation of the same [16, 17]. Indeed, the report is clearly influenced by the quality of the referral form itself, mainly in terms of the accuracy and completeness of the clinical information reported by the referring physician in the clinical question [17,18,19].

The appropriateness of a diagnostic procedure for a given individual is not always easy to determine because most recommendations, when available and if well formulated, are conditional upon many circumstances that are difficult to assess retrospectively [20]. Many attempts have been made to measure the inappropriateness of diagnostic procedures, mainly based on the assessment of “unwarranted” geographical variability in diagnostic services use, although the validity of this approach is still debated [21, 22]. Furthermore, this type of analysis can determine that a certain level of inappropriateness is present but cannot precisely identify where the problem lies.

Rates of inappropriate imaging diagnostic procedure use in the primary care setting vary considerably (from 0.2 to 99.9%) [23], even for a single procedure within the same country (i.e. from 2 to 28.5% inappropriate MRI in Canada) [24]. Given the substantial heterogeneity of the methods applied, the observed variability could possibly reflect the appreciable variation in determinants of deviation from guidelines. Bearing in mind these substantial methodological variations, overall inappropriateness of referrals in our study was similar to that reported in a Spanish cross-sectional study (31.4%) [2], but higher than that reported in Sweden (20%) [25] and substantially higher than that found in the UK (0.3%) [26] and Finland (7%) [27], which have implemented mandatory vetting of all requested radiation-related procedures.

In our study inappropriateness mostly stemmed from the mismatch of clinically meaningful questions with a recommended type of diagnostic procedure and was especially manifest for endoscopy and CT scans, while MRI was less affected by clinical inappropriateness, as already shown in other countries [28]. The narrower range of clinical indications for referral and increased GP awareness of the need for an adequate justification imposed in recent years by the economic impact of MRI prescribing [29] may explain part of the difference between MRI and the other imaging procedures considered. Apparently, the lower costs of CT and its wider availability are enough to counterbalance the higher radiation exposure, which should actually discourage unnecessary referrals.

Besides inconsistent matching between recommendations and protocols, this step in the appropriateness flow could also be influenced by the inconsistency between the clinical condition and the diagnostic test, in particular the erroneous choice between two alternative tests, for example between CT and MRI, which can be difficult for GPs to make without the guidance of a neurologist or musculoskeletal specialist. For instance, it has been reported that, despite the recommendations of clinical societies, the use of head CT for chronic headache or of spinal CT for acute non-specific low back pain without urgent symptoms is still a source of inappropriateness [26, 30].

In our study, only 2.5% of prescriptions flagged as urgent by the GPs were classified as appropriate, and with the exception of a small percentage (15%) of appropriate urgent neuro CT, inappropriateness of all other procedures under study was 100%. The reason for such a high rate of referrals automatically assigned as inappropriate is that according to the local prescription guidelines, all urgent conditions requiring endoscopy, CT or MRI are subject to urgent referral to the emergency unit for further evaluation and care, without the GP requesting any particular procedure. The exception to this is urgent requests for neuro CT in the presence of cranial trauma or signs and symptoms of transitory ischemic attack (TIA), resulting in a small percentage of appropriate neuro CT referrals. Inappropriate levels of urgency (given the test/question match, according to the agreed-on protocols) lead to allocative inefficiency and to the inappropriate use of resources.

Our study showed that the most important determinants of inappropriate prescribing were individual characteristics of the GP, while the organizational and managerial interventions had a smaller impact, even when they involved negotiating budgets for additional incentives and education, i.e. the primary care subdistrict level, or when they involved coordinating and integrating GPs’ activities through the healthcare districts. This is in line with the result that most of the variability seen is explained by the individual GPs, with only a minor part by the healthcare district or the primary care subdistrict. These finding suggests that, at least in our organizational context, managers at healthcare district level and GP coordinators at the primary care subdistrict level had limited impact on the appropriateness of diagnostic procedure prescribing, despite the considerable autonomy in budget definition and planning these two levels have. Actions individually targeting GPs, such as training and education to disseminate tools for implementing recommendations, may be the key to reduce variability at the GP level. However, our study is one of the first to assess the impact of the medical group organizational model on GPs’ performance. The consistent results across all diagnostic procedures assessed, with primary care medical groups showing a higher level of overall appropriateness and, although to a lesser degree, of clinical appropriateness, highlight the idea that new organizational models may favourably impact quality of care.

Strengths and limitations

As referral forms filled out by GPs were automatically assessed by a dedicated software, this study has intrinsic limitations. Given that the software’s ability to interpret the text is imperfect, misclassification of the consistency of the clinical question certainly occurred in some cases. However, the software proved to have substantial agreement with a manual review by an expert clinician, with kappa values over 0.6. Furthermore, the application of a single method to all referrals permitted a uniform assessment. As in the panel audit studies, the software assessment of overall appropriateness is limited to what is reported on the referral form; no direct information of a given patient’s health condition is available. Indeed, we could only assess the consistency between the clinical question as reported in the form, the requested test and the level of urgency attributed; we had no way to assess the trueness of any of the conditions reported or whether other unreported conditions could justify the choice of test and/ or the level of urgency. Therefore, we measured the consistency between what was reported and the agreed-on criteria, meaning that there could be appropriate referral forms that appear to be inappropriate because of incomplete reporting as well as inappropriate referral forms that appear to be appropriate because of incorrect reporting of conditions. Lastly, the assessment of appropriateness was based on a list of criteria created by the local professionals and was thus potentially limited by its incompleteness. This limitation is common to the vast majority of studies aiming to measure inappropriateness of diagnostic procedure prescribing using operational definitions of guidelines recommendations. In our study, we were not able to identify and assess “clinical inappropriateness”, or malpractice more generally, when formulating the clinical question in diagnostic prescriptions. Therefore, we considered incomplete, meaningless and clinical inappropriate referral forms together because a correctly filled prescription is a prerequisite for any monitoring and quality improvement system.

The sensitivity analysis, which examined the clinical appropriateness of referral forms, excluded those prescriptions that were not correctly completed (incomplete and/ or meaningless). It is worth noting that this indicator dramatically overestimated the proportion of clinically appropriate prescriptions: prescriptions that are not correctly completed or not completed at all were much more likely to be inappropriate than those that were completed and meaningful.

All these limitations mean that interpreting the prevalence of inappropriate prescriptions as an absolute measure of the quality of outpatient diagnostic procedure prescribing is not appropriate. Further, they resulted in an incorrect comparison with other studies due to the heterogeneity of the methods applied and the constrained generalisability of each of these designs. Instead, our method makes it possible to assess trends and to which conditions and characteristics of patients and GPs may favour inappropriateness.


Despite the promising increase in overall appropriateness of diagnostic procedure prescribing over the last five years in Reggio Emilia province, there is still room for improvement. Tailored interventions to increase GP compliance to guideline recommendations should be advocated. Improved compliance to protocols would reduce the number of unnecessary tests, the radiation exposure and the risk of adverse effects, thereby improving clinical practice and lowering healthcare costs. Interventions aimed at encouraging the uptake of new organizational models of care among GPs, such as primary care medical groups, could limit the effect GPs’ characteristics have on the variability of their prescribing practices.

Availability of data and materials

The datasets used during the current study are available from the corresponding author on a reasonable request.


  1. 1.

    Carter EA, Morin PE, Lind KD. Costs and Trends in Utilization of Low-value Services Among Older Adults With Commercial Insurance or Medicare Advantage. Med Care. 2017;55(11):931–9 Available from: [cited 2019 Feb 19].

    Article  Google Scholar 

  2. 2.

    Vilar-Palop J, Hernandez-Aguado I, Pastor-Valero M, Vilar J, González-Alvarez I, Lumbreras B. Appropriate use of medical imaging in two Spanish public hospitals: a cross-sectional analysis. BMJ Open. 2018;8(3):e019535 Available from: [cited 2019 Mar 4].

    Article  Google Scholar 

  3. 3.

    European Society of Radiology (ESR) ES of R. ESR concept paper on value-based radiology. Insights Imaging. 2017;8(5):447–54 Available from: [cited 2019 Feb 19].

    Article  Google Scholar 

  4. 4.

    Fine B, Dhanoa D. Imaging appropriateness criteria: why Canadian family physicians should care. Can Fam Physician. 2014;60(3):217–8 Erratum in: Can Fam Physician. 2014 May;60(5):432. PMID: 24627370; PMCID: PMC3952750. Available from:

    PubMed  PubMed Central  Google Scholar 

  5. 5.

    Remedios D, Brkljacic B, Ebdon-Jackson S, Hierath M, Sinitsyn V, Vassileva J. Collaboration, campaigns and champions for appropriate imaging: feedback from the Zagreb workshop. Insights Imaging. 2018;9(2):211–4 Available from: [cited 2019 Mar 4].

    CAS  Article  Google Scholar 

  6. 6.

    Reed MH. The Canadian Association of Radiologists Referral Guidelines. Can Assoc Radiol J. 2018;69(3):233–5 Available from: [cited 2019 Feb 26].

    Article  Google Scholar 

  7. 7.

    IAEA, European Commission. Justification of Medical Exposure in Diagnostic Imaging. In: Proceedings Series [Internet]. 2009 [cited 2019 Mar 4]. p. 190. Available from:

  8. 8.

    European Society of Radiology. Protection Radiation N° 178 Referral Guidelines for Medical Imaging Availability and Use in the European Union [Internet]. 2014 [cited 2019 Mar 4]. Available from:

  9. 9.

    Li S, Brantley E. Malpractice Liability Risk and Use of Diagnostic Imaging Services: A Systematic Review of the Literature. J Am Coll Radiol. 2015;12(12):1403–12 Available from: [cited 2019 Feb 26].

    Article  Google Scholar 

  10. 10.

    Petruzziello L, Hassan C, Alvaro D, Kohn A, Rossi Z, Zullo A, Cesaro P, Annibale B, Barca A, Di Giulio E, Giorgi Rossi P, Grasso E, Ridola L, Spada C, Costamagna G. Lazio appropriateness group. Appropriateness of the indication for colonoscopy: is the endoscopist the 'gold standard'? J Clin Gastroenterol. 2012;46(7):590–4. PMID: 22178958.

    Article  PubMed  Google Scholar 

  11. 11.

    Zorzi M, Senore C, Turrin A, Mantellini P, Visioli CB, Naldoni C, Sassoli De’ Bianchi P, Fedato C, Anghinoni E, Zappa M, Hassan C. Italian colorectal cancer screening survey group. Appropriateness of endoscopic surveillance recommendations in organised colorectal cancer screening programmes based on the faecal immunochemical test. Gut. 2016;65(11):1822–8. Epub 2015 Aug 21. PMID: 26297727.

    Article  PubMed  Google Scholar 

  12. 12.

    Hassan C, Di Giulio E, Marmo R, Zullo A, Annibale B. Appropriateness of the indication for colonoscopy: systematic review and meta-analysis. J Gastrointestin Liver Dis. 2011;20(3):279–86 PMID: 21961096.

    PubMed  Google Scholar 

  13. 13.

    Hassan C, Di Giulio E, Pickhardt PJ, Zullo A, Laghi A, Kim DH, Iafrate F, Morini S. Cost effectiveness of colonoscopy, based on the appropriateness of an indication. Clin Gastroenterol Hepatol. 2008;6(11):1231–6. PMID: 18995214.

    Article  PubMed  Google Scholar 

  14. 14.

    Harris C, Allen K, Ramsey W, King R, Green S. Sustainability in Health care by Allocating Resources Effectively (SHARE) 11: reporting outcomes of an evidence-driven approach to disinvestment in a local healthcare setting. BMC Health Serv Res. 2018;18(1):386. Published 2018 May 30.

    Article  PubMed  PubMed Central  Google Scholar 

  15. 15.

    Goldstein H. Multilevel statistical models. 2. New York: Edward Arnold; 1995.

    Google Scholar 

  16. 16.

    Wallis A, McCoubrie P. The radiology report — Are we getting the message across? Clin Radiol. 2011;66(11):1015–22 Available from: [cited 2019 Feb 26].

    CAS  Article  Google Scholar 

  17. 17.

    JML B, Weyler JJ, De Schepper AM, Parizel PM. The radiology report as seen by radiologists and referring clinicians: results of the COVER and ROVER surveys. Radiology. 2011;259(1):184–95 Available from: [cited 2019 Feb 26].

    Article  Google Scholar 

  18. 18.

    Pitman GA. Quality of referral: What information should be included in a request for diagnostic imaging when a patient is referred to a clinical radiologist? J Med Imaging Radiat Oncol. 2017;61(3):299–303 Available from: [cited 2019 Mar 4].

    Article  Google Scholar 

  19. 19.

    European Society of Radiology (ESR) ES of R. Summary of the proceedings of the international forum 2016: Imaging referral guidelines and clinical decision support - how can radiologists implement imaging referral guidelines in clinical routine? Insights Imaging. 2017;8(1):1–9 Available from: [cited 2019 Feb 19].

    Article  Google Scholar 

  20. 20.

    Sistrom CL. The appropriateness of imaging: a comprehensive conceptual framework. Radiology. 2009;251(3):637–49. PMID: 19474372.

    Article  PubMed  Google Scholar 

  21. 21.

    Keyhani S, Falk R, Bishop T, Howell E, Korenstein D. The Relationship Between Geographic Variations and Overuse of Healthcare Services. Med Care. 2012;50(3):257–61 Available from: [cited 2019 Feb 26].

    Article  Google Scholar 

  22. 22.

    Chassin MR, Kosecoff J, Park RE, Winslow CM, Kahn KL, Merrick NJ, et al. Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. JAMA. 1987;258(18):2533–7 Available from: [cited 2019 Feb 26].

    CAS  Article  Google Scholar 

  23. 23.

    O’Sullivan JW, Albasri A, Nicholson BD, Perera R, Aronson JK, Roberts N, et al. Overtesting and undertesting in primary care: a systematic review and meta-analysis. BMJ Open. 2018;8(2):e018557 Available from: [cited 2019 Jul 25].

    Article  Google Scholar 

  24. 24.

    Vanderby S, Peña-Sánchez JN, Kalra N, Babyn P. Finding the truth in medical imaging: painting the picture of appropriateness for magnetic resonance imaging in Canada. Can Assoc Radiol J. 2015;66(4):323–31. Epub 2015 Aug 12. PMID: 26277235.

    Article  PubMed  Google Scholar 

  25. 25.

    Almén A, Leitz W, Richter S. National Survey on justification of CT-examinations in Sweden [Internet]. Vol. 40. Stockholm: Swedish Radiation Safety Authority; 2009. Available from:

    Google Scholar 

  26. 26.

    Remedios D, Drinkwater K, Warwick R. National audit of appropriate imaging. Clin Radiol. 2014;69(10):1039–44.

    CAS  Article  Google Scholar 

  27. 27.

    Oikarinen H, Karttunen A, Pääkkö E, Tervonen O. Survey of inappropriate use of magnetic resonance imaging. Insights Imaging. 2013;4(5):729–33.

    Article  Google Scholar 

  28. 28.

    Bouëtté A, Karoussou-Schreiner A, Ducou Le Pointe H, Grieten M, de Kerviler E, Rausin L, et al. National audit on the appropriateness of CT and MRI examinations in Luxembourg. Insights Imaging. 2019;10(1):54. PMID: 31111303; PMCID: PMC6527721.

  29. 29.

    Solivetti FM, Guerrisi A, Salducca N, Desiderio F, Graceffa D, Capodieci G, et al. Appropriateness of knee MRI prescriptions: clinical, economic and technical issues. Radiol Med. 2016;121(4):315–22.

    CAS  Article  Google Scholar 

  30. 30.

    Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an Academic Medical Center: how critical is the need for improved decision support? J Am Coll Radiol. 2010;7(3):192–7.

    Article  Google Scholar 

Download references


We would like to thank Dr. Luigi Martino and Dr. Fabrizio Selmi of the “Clinika VAP” software company “IG Consulting (Maps Group)” for the unconditioned technical support. We would also like to thank Dr. Lina Bianconi, a general practitioner, for the insightful comments she provided on our results. Finally, we would like to thank Jacqueline M. Costa for the English language editing.



Author information




EB, MP, PGR, FV and FB conceived and designed the study. FP, EB, MP made substantial contributions to acquisition of data. MO, FV, PGR analysed data. MO, FV, PGR, GB, OD, FP interpreted the data. FV, OD, GB, MO PGR drafted the manuscript. All authors read, approved and critically reviewed the manuscript.

Corresponding author

Correspondence to Marta Ottone.

Ethics declarations

Ethics approval and consent to participate

This is a clinical audit on routinely collected administrative data. According to the Italian law, clinical audits do not require the approval of an ethics committee. (

Individual data were treated directly by the Reggio Emilia Local Health Authority (LHA), Department of Primary Care, data owner, in the carrying out of its institutional duties and for the purpose the data were collected; no specific consent to manage data was required. Managing GPs’ prescriptions to provide and/ or to reimburse prescribed tests and to monitor the quality and appropriateness of the prescription is a statutory mandate of the LHA. The General Director, legal representative of the Reggio Emilia LHA, mandated the monitoring and the auditing of the respect of the protocols here reported in the Delibera 2014/0011 del 29/01/2014, named the “Piano della performance 2013-2016” (Performance Plan 2013–2016).

Consent for publication

Not applicable as there are no individual details, image or other confidentiality related issues.

Competing interests

The study authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix Table 1.

Characteristics of a primary care medical group and a medical network in in Emilia-Romagna Region, Italy. Appendix Table 2. Multilevel logistic regression adjusted Odds Ratio and 95% CI of inconsistence criteria or test of referral forms for each diagnostic procedure by individual and physician variables: fixed effects estimates. All referral forms made by general practitioners in the province of Reggio Emilia, Italy, between 2012 and 2017 were included. Appendix Table 3. Multilevel analysis of the variability in inconsistence criteria or test of referral forms between GPs, primary care subdistrict and healthcare districts: random effects estimates. All referral forms made by general practitioners in the province of Reggio Emilia, Italy, between 2012 and 2017 were included.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Venturelli, F., Ottone, M., Pignatti, F. et al. Using text analysis software to identify determinants of inappropriate clinical question reporting and diagnostic procedure referrals in Reggio Emilia, Italy. BMC Health Serv Res 21, 103 (2021).

Download citation