Advertisement

Predicting instructed simulation and dissimulation when screening for depressive symptoms

  • Stephan GoerigkEmail author
  • Sven Hilbert
  • Andrea Jobst
  • Peter Falkai
  • Markus Bühner
  • Clemens Stachl
  • Bernd Bischl
  • Stefan Coors
  • Thomas Ehring
  • Frank Padberg
  • Nina Sarubin
Original Paper
  • 31 Downloads

Abstract

The intentional distortion of test results presents a fundamental problem to self-report-based psychiatric assessment, such as screening for depressive symptoms. The first objective of the study was to clarify whether depressed patients like healthy controls possess both the cognitive ability and motivation to deliberately influence results of commonly used screening measures. The second objective was the construction of a method derived directly from within the test takers’ responses to systematically detect faking behavior. Supervised machine learning algorithms posit the potential to empirically learn the implicit interconnections between responses, which shape detectable faking patterns. In a standardized design, faking bad and faking good were experimentally induced in a matched sample of 150 depressed and 150 healthy subjects. Participants completed commonly used questionnaires to detect depressive and associated symptoms. Group differences throughout experimental conditions were evaluated using linear mixed-models. Machine learning algorithms were trained on the test results and compared regarding their capacity to systematically predict distortions in response behavior in two scenarios: (1) differentiation of authentic patient responses from simulated responses of healthy participants; (2) differentiation of authentic patient responses from dissimulated patient responses. Statistically significant convergence of the test scores in both faking conditions suggests that both depressive patients and healthy controls have the cognitive ability as well as the motivational compliance to alter their test results. Evaluation of the algorithmic capability to detect faking behavior yielded ideal predictive accuracies of up to 89%. Implications of the findings, as well as future research objectives are discussed. Trial Registration The study was pre-registered at the German registry for clinical trials (Deutsches Register klinischer Studien, DRKS; DRKS00007708).

Keywords

Depression Assessment Faking Machine learning Response patterns 

Notes

Acknowledgements

The authors would like to acknowledge the kind assistance of Sabrina Immisch and Karoline Ziener (Department of Psychiatry and Psychotherapy, Ludwig-Maximilians-University, Munich, Germany).

Author contributions

All authors have made substantive intellectual contributions to the submitted work in form of conception of the study, and/or acquisition of data, and/or analysis and interpretation of data, and/or drafting or revising the article. NS, SH, SG, FP and AJ participated in acquisition of data and manuscript drafting; TE, PF, CS and MB participated in manuscript editing and interpretation of data; FP and NS were responsible for the concept and conduction of the study; SG, SH, BB, SC and MB were responsible for the integrity of the data and the accuracy of the data analysis; all authors approved the final version of the manuscript and take public responsibility for its content.

Funding

This study was funded by the Dr.-Karl-Wilder-Foundation (German Insurance Association, GDV, Germany). The sponsors had no role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.

Compliance with ethical standards

Conflict of interest

All authors reported no direct or indirect financial or personal relationships, interests and affiliations relevant to the subject matter of the manuscript that have occurred over the last 3 years, or that are expected in the foreseeable future.

Supplementary material

406_2018_967_MOESM1_ESM.docx (75 kb)
Supplementary material 1 (DOCX 74 KB)
406_2018_967_MOESM2_ESM.docx (15 kb)
Supplementary material 2 (DOCX 14 KB)
406_2018_967_MOESM3_ESM.docx (33 kb)
Supplementary material 3 (DOCX 32 KB)
406_2018_967_MOESM4_ESM.docx (17 kb)
Supplementary material 4 (DOCX 16 KB)

References

  1. 1.
    van Hooft EA, Born MP (2012) Intentional response distortion on personality tests: using eye-tracking to understand response processes when faking. J Appl Psychol 97:301CrossRefGoogle Scholar
  2. 2.
    Ziegler M, MacCann C, Roberts R (2011) New perspectives on faking in personality assessment. Oxford University Press, OxfordCrossRefGoogle Scholar
  3. 3.
    Viswesvaran C, Ones DS (1999) Meta-analyses of fakability estimates: implications for personality measurement. Educ Psychol Meas 59:197–210CrossRefGoogle Scholar
  4. 4.
    Boon J, Gozna L, Hall S (2008) Detecting ‘faking bad’ on the Gudjonsson suggestibility scales. Personal Individ Differ 44:263–272CrossRefGoogle Scholar
  5. 5.
    Dorsch F, Wirtz M, Strohmer J (2013) Lexikon der Psychologie (16. Aufl.). Bern HuberGoogle Scholar
  6. 6.
    Tett RP, Simonet DV (2011) Faking in personality assessment: a “multisaturation” perspective on faking as performance. Hum Perform 24:302–321CrossRefGoogle Scholar
  7. 7.
    Ziegler M, Buehner M (2009) Modeling socially desirable responding and its effects. Educ Psychol Meas 69:548–565CrossRefGoogle Scholar
  8. 8.
    Schmidt FL, Le H, Ilies R (2003) Beyond alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychol Methods 8:206CrossRefGoogle Scholar
  9. 9.
    Tourangeau R, Rasinski KA (1988) Cognitive processes underlying context effects in attitude measurement. Psychol Bull 103:299CrossRefGoogle Scholar
  10. 10.
    Marazziti D, Consoli G, Picchetti M et al (2010) Cognitive impairment in major depression. Eur J Pharmacol 626:83–86CrossRefGoogle Scholar
  11. 11.
    Austin M-P, Mitchell P, Goodwin GM (2001) Cognitive deficits in depression: possible implications for functional neuropathology. Br J Psychiatry 178:200–206CrossRefGoogle Scholar
  12. 12.
    McCullough JP Jr (2003) Treatment for chronic depression: Cognitive behavioral analysis system of psychotherapy (CBASP). Educational Publishing Foundation, DenmarkGoogle Scholar
  13. 13.
    Rehm LP (1977) A self-control model of depression. Behav Ther 8:787–804CrossRefGoogle Scholar
  14. 14.
    Derry PA, Kuiper NA (1981) Schematic processing and self-reference in clinical depression. J Abnorm Psychol 90:286CrossRefGoogle Scholar
  15. 15.
    Hollon SD, Kendall PC (1980) Cognitive self-statements in depression: development of an automatic thoughts questionnaire. Cogn Ther Res 4:383–395CrossRefGoogle Scholar
  16. 16.
    Henry GM, Weingartner H, Murphy DL (1973) Influence of affective states and psychoactive drugs on verbal learning and memory. Am J Psychiatry 130:966–971CrossRefGoogle Scholar
  17. 17.
    Bulbena A, Berrios GE (1993) Cognitive function in the affective disorders: a prospective study. Psychopathology 26:6–12CrossRefGoogle Scholar
  18. 18.
    Pauls CA, Crost NW (2005) Cognitive ability and self-reported efficacy of self-presentation predict faking on personality measures. J Individ Differ 26:194–206CrossRefGoogle Scholar
  19. 19.
    Kaplan HI, Sadock BJ (1998) Kaplan and Sadock’s synopsis of psychiatry: behavioral sciences/clinical psychiatry. Williams & Wilkins Co, BaltimoreGoogle Scholar
  20. 20.
    Smallberg M (1982) Effort and cognition in depression. Arch Gen Psychiatry 39:593–597CrossRefGoogle Scholar
  21. 21.
    Wolkenstein L, Schönenberg M, Schirm E, Hautzinger M (2011) I can see what you feel, but I can’t deal with it: impaired theory of mind in depression. J Affect Disord 132:104–111CrossRefGoogle Scholar
  22. 22.
    Wang Y, Wang Y, Chen S et al (2008) Theory of mind disability in major depression with or without psychotic symptoms: a componential view. Psychiatry Res 161:153–161CrossRefGoogle Scholar
  23. 23.
    Wilbertz G, Brakemeier E-L, Zobel I et al (2010) Exploring preoperational features in chronic depression. J Affect Disord 124:262–269CrossRefGoogle Scholar
  24. 24.
    Ziegler M (2011) Applicant faking: a look into the black box. Ind Organ Psychol 49:29–36Google Scholar
  25. 25.
    Widder B (2011) Beurteilung der Beschwerdenvalidität. Begutacht Neurol 2:64–92Google Scholar
  26. 26.
    Van Egmond J, Kummeling I, aan Balkom T (2005) Secondary gain as hidden motive for getting psychiatric treatment. Eur Psychiatry 20:416–421CrossRefGoogle Scholar
  27. 27.
    Van Egmond J, Kummeling I (2002) A blind spot for secondary gain affecting therapy outcomes. Eur Psychiatry 17:46–54CrossRefGoogle Scholar
  28. 28.
    Aronoff GM, Mandel S, Genovese E et al (2007) Evaluating malingering in contested injury or illness. Pain Pract 7:178–204CrossRefGoogle Scholar
  29. 29.
    Lieb M, Palm U, Meyer S et al (2014) Risikofaktoren für Suizide und Suizidversuche an einem Universitätsklinikum. Psychiatr Prax 41:195–199PubMedGoogle Scholar
  30. 30.
    Griffin GA, Normington J, May R, Glassmire D (1996) Assessing dissimulation among social security disability income claimants. J Consult Clin Psychol 64:1425CrossRefGoogle Scholar
  31. 31.
    Cima M, Hollnack S, Kremer K et al (2003) Strukturierter Fragebogen Simulierter Symptome. Nervenarzt 74:977–986CrossRefGoogle Scholar
  32. 32.
    Vossler-Thies E, Stevens A, Engel RR, Licha C (2013) Erfassung negativer Antwortverzerrungen mit der deutschen Fassung des “Personality Assessment Inventory”, dem “Verhaltens-und Erlebensinventar”. DiagnosticaGoogle Scholar
  33. 33.
    Stevens A, Friedel E, Mehren G, Merten T (2008) Malingering and uncooperativeness in psychiatric and psychological assessment: prevalence and effects in a German sample of claimants. Psychiatry Res 157:191–200CrossRefGoogle Scholar
  34. 34.
    Greve KW, Ord J, Curtis KL et al (2008) Detecting malingering in traumatic brain injury and chronic pain: a comparison of three forced-choice symptom validity tests. Clin Neuropsychol 22:896–918CrossRefGoogle Scholar
  35. 35.
    Donovan JJ, Dwight SA, Schneider D (2014) The impact of applicant faking on selection measures, hiring decisions, and employee performance. J Bus Psychol 29:479–493CrossRefGoogle Scholar
  36. 36.
    Deutsche R Erwerbsminderungsrente im Zeitablauf 2016. Stat Dtsch RentenversicherGoogle Scholar
  37. 37.
    Hough LM, Eaton NK, Dunnette MD et al (1990) Criterion-related validities of personality constructs and the effect of response distortion on those validities. J Appl Psychol 75:581CrossRefGoogle Scholar
  38. 38.
    Kuncel NR, Borneman MJ (2007) Toward a new method of detecting deliberately faked personality tests: the use of idiosyncratic item responses. Int J Sel Assess 15:220–231CrossRefGoogle Scholar
  39. 39.
    Holden RR, Lambert CE (2015) Response latencies are alive and well for identifying fakers on a self-report personality inventory: a reconsideration of van Hooft and Born (2012). Behav Res Methods 47:1436–1442CrossRefGoogle Scholar
  40. 40.
    Stark S, Chernyshenko OS, Chan K-Y et al (2001) Effects of the testing situation on item responding: cause for concern. J Appl Psychol 86:943CrossRefGoogle Scholar
  41. 41.
    Sjöberg L (2015) Correction for faking in self-report personality tests. Scand J Psychol 56:582–591CrossRefGoogle Scholar
  42. 42.
    Youngjohn JR, Lees-Haley PR, Binder LM (1999) Comment: warning malingerers produces more sophisticated malingering. Arch Clin Neuropsychol 14:511–515PubMedGoogle Scholar
  43. 43.
    Converse PD, Oswald FL, Imus A et al (2008) Comparing personality test formats and warnings: effects on criterion-related validity and test-taker reactions. Int J Sel Assess 16:155–169CrossRefGoogle Scholar
  44. 44.
    Heggestad ED, Morrison M, Reeve CL, McCloy RA (2006) Forced-choice assessments of personality for selection: evaluating issues of normative assessment and faking resistance. J Appl Psychol 91:9CrossRefGoogle Scholar
  45. 45.
    Komar S, Komar JA, Robie C, Taggar S (2010) Speeding personality measures to reduce faking. J Pers Psychol 9:126–137Google Scholar
  46. 46.
    Fine S, Pirak M (2016) Faking fast and slow: within-person response time latencies for measuring faking in personnel testing. J Bus Psychol 31:51–64CrossRefGoogle Scholar
  47. 47.
    Holden RR, Book AS (2009) Using hybrid Rasch-latent class modeling to improve the detection of fakers on a personality inventory. Personal Individ Differ 47:185–190CrossRefGoogle Scholar
  48. 48.
    DGPPN B, KBV A, AkdÄ B et al et al (2009) für die Leitliniengruppe Unipolare Depression (2009): S3-Leitlinie/Nationale VersorgungsLeitlinie Unipolare Depression-Langfassung, 1. Aufl DGPPN ÄZQ AWMF–Berlin DüsseldGoogle Scholar
  49. 49.
    Kohavi R, Provost F (1998) Glossary of terms. Mach Learn 30:271–274CrossRefGoogle Scholar
  50. 50.
    Russell S, Norvig P, Intelligence A (1995) A modern approach. Artif Intell Prentice-Hall Egnlewood Cliffs 25:27Google Scholar
  51. 51.
    Kotsiantis SB, Sotiris B, Zaharakis I et al (2007) Supervised machine learning: a review of classification techniques. Emerg Artif Intell Appl Comput Eng 160:3–24Google Scholar
  52. 52.
    Urbanowicz RJ, Moore JH (2009) Learning classifier systems: a complete introduction, review, and roadmap. J Artif Evol Appl 2009:1Google Scholar
  53. 53.
    Mohri M, Rostamizadeh A, Talwalkar A (2012) Foundations of machine learning. MIT press, CambridgeGoogle Scholar
  54. 54.
    Gopinathan KM, Biafore LS, Ferguson WM et al. (1998) Fraud detection using predictive modeling. U.S. Patent No. 5, 819, 226, U.S. Patent and Trademark Office, Washington, DCGoogle Scholar
  55. 55.
    Bolton RJ, Hand DJ (2002) Statistical fraud detection: a review. Stat Sci 235–249Google Scholar
  56. 56.
    Davatzikos C, Ruparel K, Fan Y et al (2005) Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroimage 28:663–668CrossRefGoogle Scholar
  57. 57.
    Ormerod T, Morley N, Ball L et al (2003) Using ethnography to design a Mass Detection Tool (MDT) for the early discovery of insurance fraud. In: CHI’03 Extended Abstracts on Human Factors in Computing Systems. ACM, pp 650–651Google Scholar
  58. 58.
    Ortega PA, Figueroa CJ, Ruz GA (2006) A medical claim fraud/abuse detection system based on data mining: a case study in chile. DMIN 6:26–29Google Scholar
  59. 59.
    Dua P, Bais S (2014) Supervised learning methods for fraud detection in healthcare insurance. In: Machine learning in healthcare informatics. Springer, Berlin, 261–285CrossRefGoogle Scholar
  60. 60.
    Cruz JA, Wishart DS (2006) Applications of machine learning in cancer prediction and prognosis. Cancer Inform 2:59CrossRefGoogle Scholar
  61. 61.
    Koutsouleris N, Meisenzahl EM, Davatzikos C et al (2009) Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition. Arch Gen Psychiatry 66:700–712CrossRefGoogle Scholar
  62. 62.
    Thrun S, Pratt L (2012) Learning to learn. Springer Science & Business Media, BerlinGoogle Scholar
  63. 63.
    Wittchen HU, Zaudig M, Fydrich T (1997) SKID-I und SKID-II. Strukt Klin Interview Für DSM-IV Hogrefe GöttGoogle Scholar
  64. 64.
    Hautzinger M, Keller F, Kühner C (2006) Beck depressions-inventar (BDI-II). Harcourt Test Services FrankfurtGoogle Scholar
  65. 65.
    Franke GH, Derogatis LR (2002) Symptom-Checkliste von LR Derogatis: SCL-90-R; deutsche Version. Beltz TestGoogle Scholar
  66. 66.
    Borkenau P, Ostendorf F (2008) NEO-FFI: NEO-Fünf-Faktoren-Inventar nach Costa und McCrae, ManualGoogle Scholar
  67. 67.
    Brickenkamp R, Schmidt-Atzert L, Liepmann D (2010) Test d2-Revision: Aufmerksamkeits-und Konzentrationstest. Hogrefe GöttingenGoogle Scholar
  68. 68.
    Dahlstrom WG, Welsh GS, Dahlstrom LE (1975) An MMPI handbook: research applications. University of Minnesota Press, MinneapolisGoogle Scholar
  69. 69.
    Greene RL (2000) The MMPI-2: an interpretive manual. Allyn & Bacon, BostonGoogle Scholar
  70. 70.
    Hiller W, Zaudig M, Mombour W (1995) ICD-10 Checklisten. Internationale Diagnosen Checklisten für ICD-10. Hans-Huber, BernGoogle Scholar
  71. 71.
    Grieve R, De Groot HT (2011) Does online psychological test administration facilitate faking? Comput Hum Behav 27:2386–2391CrossRefGoogle Scholar
  72. 72.
    Raudenbush SW, Bryk AS (2002) Hierarchical linear models: applications and data analysis methods. Sage, Thousand OaksGoogle Scholar
  73. 73.
    Bates D, Maechler M, Bolker B, Walker S (2014) lme4: linear mixed-effects models using Eigen and S4. R Package Version 1:1–23Google Scholar
  74. 74.
    Cortes C, Vapnik V (1995) Support vector machine. Mach Learn 20:273–297Google Scholar
  75. 75.
    Breiman L (2001) Random forests. Mach Learn 45:5–32CrossRefGoogle Scholar
  76. 76.
    Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38:367–378CrossRefGoogle Scholar
  77. 77.
    Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. ACM, pp 785–794Google Scholar
  78. 78.
    Bischl B, Mersmann O, Trautmann H (2010) Resampling methods in model validation. In: Workshop on Experimental Methods for the Assessment of Computational Systems (WEMACS 2010), held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN 2010), Krakow, Poland, Sept. p 14Google Scholar
  79. 79.
    Altmann A, Toloşi L, Sander O, Lengauer T (2010) Permutation importance: a corrected feature importance measure. Bioinformatics 26:1340–1347CrossRefGoogle Scholar
  80. 80.
    Bischl B, Lang M, Kotthoff L et al (2016) mlr: machine learning in R. J Mach Learn Res 17:1–5Google Scholar
  81. 81.
    Carretero-Dios H, Pérez C (2007) Standards for the development and review of instrumental studies: considerations about test selection in psychological research. Int J Clin Health Psychol 7Google Scholar
  82. 82.
    Insel T, Cuthbert B, Garvey M et al (2010) Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am Psychiatric Assoc 167:748–751Google Scholar
  83. 83.
    Dillman DA, Redline CD (2004) Testing paper self-administered questionnaires: cognitive interview and field test comparisons. Methods Test Eval Surv Quest 299–317Google Scholar
  84. 84.
    Ones DS, Viswesvaran C, Reiss AD (1996) Role of social desirability in personality testing for personnel selection: the red herring. Am Psychol Assoc 81:660–679Google Scholar
  85. 85.
    Ones DS, Viswesvaran C (1998) The effects of social desirability and faking on personality and integrity assessment for personnel selection. Hum Perform 11:245–269CrossRefGoogle Scholar
  86. 86.
    Beck AT, Steer RA, Carbin MG (1988) Psychometric properties of the Beck depression inventory: twenty-five years of evaluation. Clin Psychol Rev 8:77–100CrossRefGoogle Scholar
  87. 87.
    Bagby RM, Nicholson RA, Buis T, Bacchiochi JR (2000) Can the MMPI-2 validity scales detect depression feigned by experts? Assessment 7:55–62CrossRefGoogle Scholar
  88. 88.
    Dodd LE, Pepe MS (2003) Partial AUC estimation and regression. Biometrics 59:614–623CrossRefGoogle Scholar
  89. 89.
    Walter SD (2005) The partial area under the summary ROC curve. Stat Med 24:2025–2040CrossRefGoogle Scholar
  90. 90.
    Pannone RD (1984) Predicting test performance: a content valid approach to screening applicants. Pers Psychol 37:507–514CrossRefGoogle Scholar
  91. 91.
    Hargittai E (2009) An update on survey measures of web-oriented digital literacy. Soc Sci Comput Rev 27:130–137CrossRefGoogle Scholar
  92. 92.
    Schmidt–Atzert L, Bühner M (1998) Fehlertypen im Aufmerksamkeits-Belastungs-Test d2. DiagnosticaGoogle Scholar
  93. 93.
    Fried EI, Epskamp S, Nesse RM et al (2016) What are’good’depression symptoms? Comparing the centrality of DSM and non-DSM symptoms of depression in a network analysis. J Affect Disord 189:314–320CrossRefGoogle Scholar
  94. 94.
    Messick S (1988) Meaning and values in test validation: the science and ethics of assessment. ETS Res Rep Ser 1988Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Stephan Goerigk
    • 1
    • 2
    • 3
    Email author return OK on get
  • Sven Hilbert
    • 1
    • 4
  • Andrea Jobst
    • 2
  • Peter Falkai
    • 2
  • Markus Bühner
    • 1
  • Clemens Stachl
    • 1
  • Bernd Bischl
    • 5
  • Stefan Coors
    • 5
  • Thomas Ehring
    • 6
  • Frank Padberg
    • 2
  • Nina Sarubin
    • 1
    • 2
    • 3
  1. 1.Department of Psychological Methodology and AssessmentLudwig-Maximilians-University MunichMunichGermany
  2. 2.Department of Psychiatry and PsychotherapyLudwig-Maximilians-UniversityMunichGermany
  3. 3.Hochschule FreseniusUniversity of Applied SciencesMunichGermany
  4. 4.Faculty of Psychology, Educational Science and Sport ScienceUniversity of RegensburgRegensburgGermany
  5. 5.Department of StatisticsLudwig-Maximilians-UniversityMunichGermany
  6. 6.Department of Clinical Psychology and PsychotherapyLudwig-Maximilians-UniversityMunichGermany

Personalised recommendations