Advertisement

Theoretical Considerations and Development of a Questionnaire to Measure Trust in Automation

  • Moritz Körber
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 823)

Abstract

The increasing number of interactions with automated systems has sparked the interest of researchers in trust in automation because it predicts not only whether but also how an operator interacts with an automation. In this work, a theoretical model of trust in automation is established and the development and evaluation of a corresponding questionnaire (Trust in Automation, TiA) are described.

Building on the model of organizational trust by Mayer et al. (1995) and the theoretical account by Lee and See (2004), a model for trust in automation containing six underlying dimensions was established. Following a deductive approach, an initial set of 57 items was generated. In a first online study, these items were analyzed and based on the criteria item difficulty, standard deviation, item-total correlation, internal consistency, overlap with other items in content, and response quote, 40 items were eliminated and two scales were merged, leaving six scales (Reliability/Competence, Understandability/Predictability, Propensity to Trust, Intention of Developers, Familiarity, and Trust in Automation) containing a total of 19 items.

The internal structure of the resulting questionnaire was analyzed in a subsequent second online study by means of an exploratory factor analysis. The results show sufficient preliminary evidence for the proposed factor structure and demonstrate that further pursuit of the model is reasonable but certain revisions may be necessary. The calculated omega coefficients indicated good to excellent reliability for all scales. The results also provide evidence for the questionnaire’s criterion validity: Consistent with the expectations, an unreliable automated driving system received lower trust ratings as a reliably functioning system. In a subsequent empirical driving simulator study, trust ratings could predict reliance on an automated driving system and monitoring in form of gaze behavior. Possible steps for revisions are discussed and recommendations for the application of the questionnaire are given.

Keywords

Trust in automation Automated driving Questionnaire 

References

  1. Ajzen I, Fishbein M (1980) Understanding attitudes and predicting social behavior. Prentice Hall, Englewood CliffsGoogle Scholar
  2. Annett J (2002) Subjective rating scales: science or art? Ergonomics 45:966–987.  https://doi.org/10.1080/00140130210166951CrossRefGoogle Scholar
  3. Bagozzi RP (1982) The role of measurement in theory construction and hypothesis testing: toward a holistic model. In: Fornell C (ed) A second generation of multivariate analysis. Praeger, New York, pp 5–23Google Scholar
  4. Barber B (1983) The logic and limits of trust. Rutgers University Press, New BrunswickGoogle Scholar
  5. Beggiato M, Pereira M, Petzoldt T, Krems JF (2015) Learning and development of trust, acceptance and the mental model of ACC. A longitudinal on-road study. Transp Res Part F Traffic Psychol Behav 35:75–84.  https://doi.org/10.1016/j.trf.2015.10.005CrossRefGoogle Scholar
  6. Bisantz AM, Seong Y (2001) Assessment of operator trust in and utilization of automated decision-aids under different framing conditions. Int J Ind Ergon 28:85–97.  https://doi.org/10.1016/S0169-8141(01)00015-4CrossRefGoogle Scholar
  7. Blomqvist K (1997) The many faces of trust. Scand J Manag 13:271–286.  https://doi.org/10.1016/S0956-5221(97)84644-1CrossRefGoogle Scholar
  8. Brown RD, Galster SM (2004) Effects of reliable and unreliable automation on subjective measures of mental workload, situation awareness, trust and confidence in a dynamic flight task. In: Proceedings of the human factors and ergonomics society annual meeting 2004, pp 147–151Google Scholar
  9. Burisch M (1978) Construction strategies for multiscale personality inventories. Appl Psychol Measur 2:97–111.  https://doi.org/10.1177/014662167800200110CrossRefGoogle Scholar
  10. Burisch M (1984) Approaches to personality inventory construction: a comparison of merits. Am Psychol 39:214–227.  https://doi.org/10.1037/0003-066X.39.3.214CrossRefGoogle Scholar
  11. Butler JK, Cantrell RS (1984) A behavioral decision theory approach to modeling dyadic trust in superiors and subordinates. Psychol Rep 55:19–28.  https://doi.org/10.2466/pr0.1984.55.1.19CrossRefGoogle Scholar
  12. Butler JK (1991) Toward understanding and measuring conditions of trust: evolution of a conditions of trust inventory. J Manag 17:643–663.  https://doi.org/10.1177/014920639101700307CrossRefGoogle Scholar
  13. Damasio AR (1996) The somatic marker hypothesis and the possible functions of the prefrontal cortex. Philos Trans R Soc Lond B Biol Sci 351:1413–1420.  https://doi.org/10.1098/rstb.1996.0125CrossRefGoogle Scholar
  14. Deutsch M (1958) Trust and suspicion. J Conflict Resolut 2:265–279.  https://doi.org/10.1177/002200275800200401CrossRefGoogle Scholar
  15. Deutsch M (1960) The effect of motivational orientation upon trust and suspicion. Hum Relations 13:123–139.  https://doi.org/10.1177/001872676001300202CrossRefGoogle Scholar
  16. Diamantopoulos A, Sarstedt M, Fuchs C, Wilczynski P, Kaiser S (2012) Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective. J Acad Mark Sci 40:434–449.  https://doi.org/10.1007/s11747-011-0300-3CrossRefGoogle Scholar
  17. Drnec K, Marathe AR, Lukos JR, Metcalfe JS (2016) From trust in automation to decision neuroscience: applying cognitive neuroscience methods to understand and improve interaction decisions involved in human automation interaction. Front Hum Neurosci 10:54.  https://doi.org/10.3389/fnhum.2016.00290CrossRefGoogle Scholar
  18. Eisinga R, Grotenhuis MT, Pelzer B (2013) The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? Int J Public Health 58:637–642.  https://doi.org/10.1007/s00038-012-0416-3CrossRefGoogle Scholar
  19. Emons WHM, Sijtsma K, Meijer RR (2007) On the consistency of individual classification using short scales. Psychol Methods 12:105–120.  https://doi.org/10.1037/1082-989X.12.1.105CrossRefGoogle Scholar
  20. Feuerberg BV, Bahner JE, Manzey D (2005) Interindividuelle unterschiede im umgang mit automation – entwicklung eines fragebogens zur erfassung des complacency-potentials. In: Urbas L, Steffens C (eds) Zustandserkennung und systemgestaltung. 6. Berliner werkstatt mensch-maschine-systeme., Als Ms. gedr. VDI-Verlag, Düsseldorf, pp 199–202Google Scholar
  21. Flake JK, Pek J, Hehman E (2017) Construct validation in social and personality research. Soc Psychol Pers Sci 8:370–378.  https://doi.org/10.1177/1948550617693063CrossRefGoogle Scholar
  22. Fuchs C, Diamantopoulos A (2009) Using single-item measures for construct measurement in management research: conceptual issues and application guidelines. Die Betriebswirtschaft 69:195Google Scholar
  23. Gigerenzer G, Selten R (eds) (2002) Bounded rationality: the adaptive toolbox. MIT Press, CambridgeGoogle Scholar
  24. Heene M, Hilbert S, Draxler C, Ziegler M, Bühner M (2011) Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices. Psychol Methods 16:319–336.  https://doi.org/10.1037/a0024917CrossRefGoogle Scholar
  25. Hergeth S, Lorenz L, Vilimek R, Krems JF (2016) Keep your scanners peeled: gaze behavior as a measure of automation trust during highly automated driving. Hum Factors 58:509–519.  https://doi.org/10.1177/0018720815625744CrossRefGoogle Scholar
  26. Hergeth S, Lorenz L, Krems JF (2017) Prior familiarization with takeover requests affects drivers’ takeover performance and automation trust. Hum Factors 59:457–470.  https://doi.org/10.1177/0018720816678714CrossRefGoogle Scholar
  27. Hoeppner BB, Kelly JF, Urbanoski KA, Slaymaker V (2011) Comparative utility of a single-item versus multiple-item measure of self-efficacy in predicting relapse among young adults. J Subst Abuse Treat 41:305–312.  https://doi.org/10.1016/j.jsat.2011.04.005CrossRefGoogle Scholar
  28. Hoff KA, Bashir M (2015) Trust in automation: integrating empirical evidence on factors that influence trust. Hum Factors 57:407–434.  https://doi.org/10.1177/0018720814547570CrossRefGoogle Scholar
  29. Hoffman RR, Johnson M, Bradshaw JM, Underbrink A (2013) Trust in automation. IEEE Intell Syst 28:84–88.  https://doi.org/10.1109/MIS.2013.24CrossRefGoogle Scholar
  30. Horn JL (1965) A rationale and test for the number of factors in factor analysis. Psychometrika 30:179–185.  https://doi.org/10.1007/BF02289447CrossRefzbMATHGoogle Scholar
  31. Hu L-T, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Eq Model Multidiscip J 6:1–55.  https://doi.org/10.1080/10705519909540118CrossRefGoogle Scholar
  32. Hubley AM, Zumbo BD (2013) Psychometric characteristics of assessment procedures: An overview. In: Geisinger KF (ed) Test theory and testing and assessment in industrial and organizational psychology, 1st edn. American Psychological Association, Washington, pp 3–20Google Scholar
  33. Jian J-Y, Bisantz AM, Drury CG (2000) Foundations for an empirically determined scale of trust in automated systems. Int J Cogn Ergon 4:53–71.  https://doi.org/10.1207/S15327566IJCE0401_04CrossRefGoogle Scholar
  34. Kenny DA, Kaniskan B, McCoach DB (2014) The performance of RMSEA in models with small degrees of freedom. Soc Methods Res 44:486–507.  https://doi.org/10.1177/0049124114543236MathSciNetCrossRefGoogle Scholar
  35. Kirlik A (1993) Modeling strategic behavior in human-automation interaction: why an “aid” can (and should) go unused. Hum Factors 35:221–242.  https://doi.org/10.1177/001872089303500203CrossRefGoogle Scholar
  36. Körber M (2018) Individual differences in human-automation interaction: a driver-centered perspective on the introduction of automated vehicles. Dissertation, Technical University of MunichGoogle Scholar
  37. Körber M, Baseler E, Bengler K (2018) Introduction matters: manipulating trust in automation and reliance in automated driving. Appl Ergon 66:18–31.  https://doi.org/10.1016/j.apergo.2017.07.006CrossRefGoogle Scholar
  38. Krantz DH, Luce RD, Suppes P, Tversky A (2007) Additive and polynomial representations. Foundations of measurement, vol 1. Dover Publisher, MineolaGoogle Scholar
  39. Lee JD, Moray N (1992) Trust, control strategies and allocation of function in human-machine systems. Ergonomics 35:1243–1270.  https://doi.org/10.1080/00140139208967392CrossRefGoogle Scholar
  40. Lee JD, See KA (2004) Trust in automation: designing for appropriate reliance. Hum Factors 46:50–80.  https://doi.org/10.1518/hfes.46.1.50_30392CrossRefGoogle Scholar
  41. Love J, Selker R, Verhagen J, Marsman M, Gronau QF, Jamil T, Šmíra M, Epskamp S, Wild A, Ly A, Matzke D, Wagenmakers E-J, Morey RD, Rouder JN (2015) Software to sharpen your stats. APS Obs 28:27–29Google Scholar
  42. MacCallum RC, Browne MW, Sugawara HM (1996) Power analysis and determination of sample size for covariance structure modeling. Psychol Methods 1:130–149.  https://doi.org/10.1037/1082-989X.1.2.130CrossRefGoogle Scholar
  43. MacCallum RC, Widaman KF, Zhang S, Hong S (1999) Sample size in factor analysis. Psychol Methods 4:84–99.  https://doi.org/10.1037/1082-989X.4.1.84CrossRefGoogle Scholar
  44. Madhavan P, Wiegmann DA (2007) Similarities and differences between human–human and human–automation trust: an integrative review. Theor Issues Ergon Sci 8:277–301.  https://doi.org/10.1080/14639220500337708CrossRefGoogle Scholar
  45. Madsen M, Gregor S (2000) Measuring human-computer trust. In: Proceedings of the 11th Australasian conference on information systems, pp 6–8Google Scholar
  46. Mayer RC, Davis JH, Schoorman FD (1995) An integrative model of organizational trust. Acad Manag Rev 20:709–734.  https://doi.org/10.5465/AMR.1995.9508080335CrossRefGoogle Scholar
  47. Mayer RC, Davis JH (1999) The effect of the performance appraisal system on trust for management: a field quasi-experiment. J Appl Psychol 84:123–136.  https://doi.org/10.1037//0021-9010.84.1.123CrossRefGoogle Scholar
  48. McCoach DB (2003) SEM isn’t just the Schoolwide enrichment model anymore: structural equation modeling (SEM) in gifted education. J Educ Gifted 27:36–61.  https://doi.org/10.1177/016235320302700104CrossRefGoogle Scholar
  49. McCoach DB, Gable RK, Madura JP (2013) Instrument development in the affective domain: School and corporate applications, 3rd edn. Springer, New YorkCrossRefGoogle Scholar
  50. McKnight DH, Chervany NL (2001) Trust and distrust definitions: one bite at a time. In: Falcone R, Singh M, Tan Y-H (eds) Trust in cyber-societies: integrating the human and artificial perspectives. Springer, Heidelberg, pp 27–54CrossRefGoogle Scholar
  51. McNeish D (2017) Thanks coefficient alpha, we’ll take it from here. Psychol Methods.  https://doi.org/10.1037/met0000144CrossRefGoogle Scholar
  52. Merritt SM, Ilgen DR (2008) Not all trust is created equal: dispositional and history-based trust in human-automation interactions. Hum Factors 50:194–210.  https://doi.org/10.1518/001872008X288574CrossRefGoogle Scholar
  53. Meyer J (2004) Conceptual issues in the study of dynamic hazard warnings. Hum Factors 46:196–204CrossRefGoogle Scholar
  54. Michell J (1997) Quantitative science and the definition of measurement in psychology. Br J Psychol 88:355–383.  https://doi.org/10.1111/j.2044-8295.1997.tb02641.xCrossRefGoogle Scholar
  55. Moosbrugger H, Kelava A (eds) (2012) Testtheorie und fragebogenkonstruktion, 2nd edn. Springer, HeidelbergGoogle Scholar
  56. Muir BM (1987) Trust between humans and machines, and the design of decision aids. Int J Man Mach Stud 27:527–539.  https://doi.org/10.1016/S0020-7373(87)80013-5CrossRefGoogle Scholar
  57. Muir BM (1994) Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics 37:1905–1922.  https://doi.org/10.1080/00140139408964957CrossRefGoogle Scholar
  58. Muir BM, Moray N (1996) Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics 39:429–460.  https://doi.org/10.1080/00140139608964474CrossRefGoogle Scholar
  59. Nagy MS (2002) Using a single-item approach to measure facet job satisfaction. J Occup Organ Psychol 75:77–86.  https://doi.org/10.1348/096317902167658CrossRefGoogle Scholar
  60. Nunnally JC, Bernstein IH (1994) Psychometric theory, 3rd edn. McGraw-Hill, New YorkGoogle Scholar
  61. Parasuraman R, Sheridan TB, Wickens CD (2000) A model for types and levels of human interaction with automation. IEEE Trans Syst Man Cybern 30:286–297.  https://doi.org/10.1109/3468.844354CrossRefGoogle Scholar
  62. Paulhus DL, Vazire S (2007) The self-report method. In: Robins RW, Fraley RC, Krueger RF (eds) Handbook of research methods in personality psychology. The Guilford Press, New York, pp 224–239Google Scholar
  63. Preacher KJ, Zhang G, Kim C, Mels G (2013) Choosing the optimal number of factors in exploratory factor analysis: a model selection perspective. Multivar Behav Res 48:28–56.  https://doi.org/10.1080/00273171.2012.710386CrossRefGoogle Scholar
  64. Rempel JK, Holmes JG, Zanna MP (1985) Trust in close relationships. J Pers Soc Psychol 49:95–112.  https://doi.org/10.1037/0022-3514.49.1.95CrossRefGoogle Scholar
  65. Rhemtulla M, Brosseau-Liard PÉ, Savalei V (2012) When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychol Methods 17:354–373.  https://doi.org/10.1037/a0029315CrossRefGoogle Scholar
  66. Robins RW, Hendin HM, Trzesniewski KH (2001) Measuring global self-esteem: construct validation of a single-item measure and the Rosenberg self-esteem scale. Pers Soc Psychol Bull 27:151–161.  https://doi.org/10.1177/0146167201272002CrossRefGoogle Scholar
  67. Rotter JB (1971) Generalized expectancies for interpersonal trust. Am Psychol 26:443–452.  https://doi.org/10.1037/h0031464CrossRefGoogle Scholar
  68. Sakaluk JK, Short SD (2017) A methodological review of exploratory factor analysis in sexuality research: used practices, best practices, and data analysis resources. J Sex Res 54:1–9.  https://doi.org/10.1080/00224499.2015.1137538CrossRefGoogle Scholar
  69. Scarpello V, Campbell JP (1983) Job satisfaction: are all the parts there? Pers Psychol 36:577–600.  https://doi.org/10.1111/j.1744-6570.1983.tb02236.xCrossRefGoogle Scholar
  70. Sloan JA, Aaronson N, Cappelleri JC, Fairclough DL, Varricchio C (2002) Assessing the clinical significance of single items relative to summated scores. Mayo Clin Proc 77:479–487.  https://doi.org/10.4065/77.5.479CrossRefGoogle Scholar
  71. Tal E (2017) Measurement in science. In: Zalta EN (ed) The Stanford encyclopedia of philosophy. Metaphysics Research Lab, Stanford UniversityGoogle Scholar
  72. Uebersax JS (2006) Likert scales: dispelling the confusion. http://www.john-uebersax.com/stat/likert.htm. Accessed 8 Feb 2018

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Chair of ErgonomicsTechnical University of MunichGarchingGermany

Personalised recommendations