Quantifying Analysis of Uncertainty in Medical Reporting: Creation of User and Context-Specific Uncertainty Profiles

Article

Abstract

While uncertainty is ubiquitous in medical practice, minimal work to date has been performed to analyze the cause and effect relationship between uncertainty and patient outcomes. In medical imaging practice, uncertainty in the radiology report has been well documented to be a source of clinician dissatisfaction. Before one can effectively create intervention strategies aimed at reducing uncertainty, it must first be better understood through context- and user-specific analysis. One strategy for accomplishing this task is to characterize the source of uncertainty and create user-specific uncertainty profiles which take into account a number of provider-specific variables which may contribute to report uncertainty. The resulting data can in turn be used to create real-time report uncertainty metrics aimed at providing uncertainty analytics at the point of care, for the combined purposes of decision support, improved communication, and enhanced clinical/economic outcomes.

Keywords

Uncertainty Data mining Profiling Outcomes analysis 

Introduction

While uncertainty in medical reporting has been well documented to have a negative influence on clinical outcomes [1, 2], little research to date has been performed to evaluate how the perceptions and actions related to medical uncertainty affect individual healthcare providers and user groups. Before creating an innovation strategy aimed at both decreasing uncertainty frequency and improved understanding of its use, one must first create an objective methodology of uncertainty analysis which can be applied to all text-based forms of medical communication with the ability to create standardized quantitative and qualitative uncertainty metrics [3]. If successfully implemented, these uncertainty metrics can be used to populate a standardized referenceable database which can comingle large number of medical report data from a variety of institutional and individual providers. With large sample size statistics, the derived analytics can take into account the occupational setting, clinical context, type of medical report, medical data being used (both historical and current), contributing technology, and attributes specific to both the patient and provider.

The correlation of uncertainty data with attributes of the individual provider and patient can in turn lead to the creation of uncertainty profiles, which can provide valuable insights related to contributing user-specific factors (i.e., confounding variables) which affect uncertainty usage in reporting, and the potential impact uncertainty plays in clinical outcomes.

Theoretical Considerations to End-User Profiling

Profiling has been defined as the act or process of extrapolating information about a person based on known traits or tendencies (i.e., past behavior) (https://www.merriam-webster.com/dictionary/profiling), so as to assess or predict their actions in accordance with identifying characteristics within a particular subgroup. One well-known application is criminal profiling, which is an investigative tool used by law enforcement to identify likely suspects and analyze patterns which may predict future actions [4]. Other less well-known applications include linguistic and author profiling, which can be applied to both speech and text and used to analyze both word content and style [5, 6].

As it pertains to criminal profiling, five steps have been described in the creation of profiles [7].
  1. 1.

    Analyze the criminal act.

     
  2. 2.

    Compare it to similar crimes.

     
  3. 3.

    Perform in-depth analysis of the actual crime.

     
  4. 4.

    Consideration of the victim’s background and outcomes for possible connections.

     
  5. 5.

    Develop a description of the offender that can be compared with previous cases.

     
If one was to apply these principles to the proposed creation of medical end-user profiles, the following steps could be defined:
  1. 1.

    Analyze the report content (e.g., frequency and severity of uncertainty, specific language used).

     
  2. 2.

    Compare it to similar actions (e.g., report type, clinical context, historical uncertainty usage).

     
  3. 3.

    Perform in-depth analysis of the event (e.g., medical data quality and quantity, patient attributes, technology).

     
  4. 4.

    Consideration of the end-user’s background (e.g., education/training, practice type, clinical experience).

     
  5. 5.

    Develop a description of the end user (e.g., uncertainty tendencies, decision support use, follow-up recommendations)

     

The net effect is that the same principles used for criminal profiling can be applied to medical end-user profiling. In this application, medical report content becomes the subject matter, and standardized text-based analysis becomes the medium with which data is derived and analyzed in accordance with context and user-specific variables.

Characterizing Uncertainty

Report uncertainty can manifest itself in a number of ways. Some uncertainty can be thought of as expected or predictable, based upon the presence of contributing or mitigating factors which are often seen in association with report uncertainty (Table 1) [8]. Collectively, these amount to a variety of factors which may adversely affect quality, completeness, appropriateness, accuracy, or availability of clinical and/or imaging data. When data is compromised in some form or fashion, one can expect some degree of concomitant uncertainty, which may be manifested in the report in a global or finding-specific fashion. For radiology reporting, commonly encountered examples include poor image quality (e.g., motion artifact), improper protocol selection, deficient clinical data, limitations in technology, detrimental patient attributes (e.g., morbid obesity), complex imaging data, unavailable historical imaging data, clinical diagnosis poorly established through medical imaging, heightened medico-legal risk, complex anatomy, and lack of established diagnostic standards.
Table 1

Contributing factors to uncertainty in radiology reporting

1. Clinical data

2.Exam and protocol selection

3. Image quality

4. Technology in use

5. Patient profile

6. Exam complexity

7. Historical imaging data

8. Individual finding or disease

9. Medicolegal risk

10. Radiologist profile

11. Anatomic variation

12. Community/societal standards

In other circumstances, report uncertainty cannot be accurately predicted based upon contributing factors and would be characterized as unexpected or idiosyncratic. These instances of unexpected uncertainty can be thought of as unique to the individual radiologist. In some circumstances, their presence may be repetitive and predictable, as in the example where a given radiologist tends to repeatedly (and predictably) qualify findings related to the appendix on all pediatric abdominal CT exams. In other cases, instances of uncertainty may be sporadic and unpredictable, which can be classified as uncertainty outliers. The differentiation of expected and unexpected uncertainty may prove to be important to the referring clinician in understanding the context of the uncertainty as well as determining the appropriate course of action.

As an example, take a pediatric extremity x-ray report performed in the evaluation of trauma, which contains the following language. “While no fracture is currently visualized, follow-up radiographs in 7 days is recommended if an occult fracture remains of concern”. In isolation (and the absence of historical data analysis), the referring clinician would in likely order follow-up radiographs due to the inherent uncertainty contained in the report. If however, the clinician was presented with context and user-specific uncertainty data, they could make a data-driven decision as to the need for follow-up radiographs. In one example, suppose the authoring radiologist is shown to incorporate this uncertainty language into 90% of all pediatric x-ray reports for trauma, with a positive predictive value of 5%. In another example, the authoring radiologist incorporates this uncertainty language in 15% of similar reports, with a positive predictive value of 50%. Having objective data-specific to the end user and context could assist in the clinician in determining the relative clinical significance of the uncertainty language in question, which could in theory lead to improved clinical and economic outcomes.

In addition to characterizing uncertainty by its predictability, uncertainty should be analyzed in accordance with its frequency, manner (i.e., specific terminology), context, and severity (https://www.merriam-webster.com/dictionary/profiling). Collectively, these uncertainty characteristics formulate the basis with which uncertainty can be defined, quantified, and analyzed, with the goals of improved report communication and understanding.

Creation of Uncertainty Profiles

The concept of creating profiles is centered on the idea that healthcare provider groups are inherently heterogeneous in nature and can be subdivided into segmented populations based upon common traits, which in turn can be used to predict practice patterns and performance outcomes. When applied to uncertainty in reporting, these profiles are intended to analyze user- and context-specific uncertainty, with the goal of improved clinical outcomes.

Questions to ask:
  1. 1.

    What constitutes “expected” uncertainty use versus unexpected uncertainty?

     
  2. 2.

    To what degree does uncertainty differ between providers performing the same task?

     
  3. 3.

    What factors contribute to uncertainty and do they affect all providers to the same degree?

     
  4. 4.

    Is it possible to predict uncertainty a priori based on historical data analysis specific to the individual provider and clinical context?

     
  5. 5.

    How does uncertainty relate to clinical outcomes? Do some providers utilize uncertainty more efficiently and accurately (as it relates to clinical outcomes) than others?

     
  6. 6.

    How can user and context specific uncertainty analyses be used to create customizable decision support tools for improved communication, understanding, and clinical outcomes?

     
Table 2 provides a list of variables which can be used to define an individual end-user’s uncertainty profile, which is comprised of three separate sections. The first section consists of variables which define the professional and personal attributes of the individual person, containing both fixed and dynamic variables. Fixed variables (e.g., practice type) remain relatively constant over time, while dynamic variables (e.g., stress) require periodic adjustment to account for interval change. In totality, these variables attempt to define an end-user’s similarities and differences relative to his or her peer group.
Table 2

Provider profile

A. Professional and personal attributes

B. Report components

C. Workflow

A. Professional and personal

 1. Demographics

 2. Clinical experience

 3. Education and training

 4. Practice type

 5. Personality

 6. Peer review and medicolegal history

 7. Technology utilization and experience

 8. Physical state

 9. Emotional state

B. Reporting

 1. General usage

 2. Exam/test specific

 3. Workflow

 4. Outcomes analysis

 5. Data availability and quality

 6. Decision support utilization

 7. Report context

 8. Patient profile

 9. Perceptual differences

C. Workflow

 1. Workload/productivity

 2. Task complexity

 3. Backlog

 4. Exam/procedural time

 5. Interruptions

 6. Prioritization status

The second section of the end-user profile consists of report uncertainty metrics (both qualitative and quantitative) and analyses. Quantitative metrics include frequency and severity of uncertainty, while qualitative metrics include the specific terms used and the clinical context in which they are applied. In many use cases, contributing factors are present which may be directly or indirectly related to uncertainty. Examples may include insufficient clinical data, patient noncompliance, or poor data quality.

The third category of report uncertainty relates to workflow which can include a number of variables including task complexity, cumulative workload, test/procedural time, backlog, interruptions, and task prioritization. Any of these factors may be shown to affect report uncertainty to varying degrees, in accordance with the individual provider. As an example, one end user may respond to increased exam complexity or volume by introducing increased uncertainty severity or frequency, while another end user may be relatively unaffected by changes in workload. At the same time, the end user affected by workload change may demonstrate uncertainty change only for specific exam types or individual findings. This illustrates the dynamic nature of report uncertainty, which requires multivariate analysis. The various patterns derived through these analyses can be used to define different profile groups within a given provider category, while also providing valuable insights relating to decision support and other interventional strategies.

Context and User-Specific Uncertainty Predicative Analytics

With the assistance of NLP and artificial intelligence (e.g., machine learning), uncertainty data can be prospectively recorded and analyzed over time to create a user-specific uncertainty profile. Taking into account a number of exam-specific variables (e.g., modality, exam type, clinical indication), one could in effect predict the frequency, context, and severity of report uncertainty based upon historical uncertainty data of the individual end user.

As the radiologist generates a report for the current exam, these historical predicative analytics can be directly correlated with real-time uncertainty data in the current report analysis. If higher than expected uncertainty data is recorded in the current report analysis, an automated prompt could alert the radiologist to uncertainty exceeding their predefined threshold and the potential for an uncertainty outlier/s requiring more in-depth analysis. To illustrate how this may work, we can take the scenario where a radiologist is reading a chest CT angiography study for suspected pulmonary emboli. Based upon historical analysis of the radiologist, the following uncertainty data is retrieved:
  • Average number of uncertainty terms per chest CTA for “pulmonary emboli” report: 3.3

  • Average uncertainty severity score: 0.28

  • Most common uncertainty findings: pulmonary embolus, nodule, pleural effusion

In the course of current report analysis, a higher than expected uncertainty score of 0.4 is identified in association with the finding of pulmonary embolus, Historical review for radiologist uncertainty specific to this finding demonstrates an average severity score of 0.28. This higher than expected uncertainty score would result in the designation of an uncertainty outlier. This could in turn generate an automated prompt alerting the radiologist, who may elect to modify report language, utilize decision support (e.g., CAD, advanced visualization), or request consultation with a colleague. In addition, the designation of the uncertainty outlier may also invoke automated radiologist feedback of follow-up imaging and/or clinical testing along with outcomes analysis.

The intended purpose of this combined uncertainty and profile analysis is to provide real-time analysis and feedback during report creation to assist in the identification of unexpected uncertainty outliers which are specific to the individual end user, exam type, and clinical context. By doing so, the goal is to facilitate targeted intervention at the point of care with the hope of improved diagnostic confidence, report understanding, and clinical outcomes.

References

  1. 1.
    Carney PA, Yi JP, Abraham LA et al.: Reactions to uncertainty and the accuracy of diagnostic mammography. J Gen Intern Med 22:234–241, 2007CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Wang SY, Chan WP: Uncertainty and its consequences in clinical practice. J Korean Med Sci 30:1710–1712, 2015CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Reiner B: Quantitative analysis of uncertainty in medical reporting. J Digit Imaging 2017 (Submitted for publication)Google Scholar
  4. 4.
    Woodhams J, Toye K: An empirical test of the assumptions of case linkage and offender profiling with serial commercial robberies. Psychology, Public Policy, and Law 13:59–85, 2007CrossRefGoogle Scholar
  5. 5.
    Argamon S, Koppel M: A systematic functional approach to automated authorship analysis. Journal of law and Policy 21:299–316, 2013Google Scholar
  6. 6.
    Van Halteren H: Linguistic profiling for authorship recognition and verification. In: Proceedings of the 42nd conference of the ACL. East Stroudsburg: PA:ACL, pp. 199–206Google Scholar
  7. 7.
    Douglas JE, Burgess AE: Criminal profiling: a viable investigative tool against violent crime. FBL Law Enforcement Bulletin 55:9–13, 1986Google Scholar
  8. 8.
    Reiner B: Contextualizing causation of uncertainty in medical reporting. J Am Coll Radiol 2017 (in press)Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  1. 1.Maryland VA Healthcare SystemBaltimoreUSA

Personalised recommendations