Towards Scalable, Integrative Assessment of Children’s Self-Regulatory Capabilities: New Applications of Digital Technology
The assessment of self-regulation in children is of significant interest to researchers within education, clinical and developmental psychology, and clinical neuroscience, given its importance to adaptive functioning across a wide range of social, educational, interpersonal, educational and health domains. Because self-regulation is a complex, multidimensional construct, a range of assessment approaches have been developed to assess its various components including behavioural, cognitive and emotional domains. In recent years, digital technology has been increasingly used to enhance or supplement existing measurement approaches; however, developments have predominantly focused on translating traditional testing paradigms into digital formats. There is a need for more innovation in digital psychological assessments that harness modern mechanisms such as game-based design and interactivity. Such approaches have potential for the development of scalable, adaptable universal approaches to screening and assessment of children’s self-regulatory capabilities, to facilitate early identification of difficulties in individuals and also guide planning and decision-making at a population level. We highlight a novel, innovative digital assessment tool for children called Rumble’s Quest, a new measure of children’s socio-emotional functioning that shows promise as an integrative assessment of well-being and self-regulation, and which incorporates both self-report and direct assessment of cognitive self-regulation. This tool is scalable, can be integrated into normal classroom activities, and forms part of a comprehensive prevention support system that can be used to guide stakeholders’ decision-making regarding early intervention and support at the individual, classroom, school and community level. We finish by discussing other innovative possibilities for psychological assessment with children, using new and emerging technologies and assessment approaches.
KeywordsSelf-regulation Psychological assessment Digital technology Children
Compliance with Ethical Standards
Conflict of interest
Authors Jamin Day and Alan Hayes have no conflicts of interest to declare. Authors Kate Freiberg and Ross Homel are co-developers of the Rumble’s Quest tool and associated digital support platforms (e.g. https://www.realwell.org.au). These authors receive no direct funding compensation or royalties. Ongoing development, validation and evaluation studies for Rumble’s Quest have been supported through Australian Research Council grants. No funding was received towards the preparation of this manuscript.
This article does not contain any studies with human participants or animals performed by any of the authors.
- Andersson, G., Bergström, J., Buhrman, M., Carlbring, P., Holländare, F., Kaldo, V., … Waara, J. (2008). Development of a new approach to guided self-help via the internet: The Swedish experience. Journal of Technology in Human Services, 26(2–4), 161–181. https://doi.org/10.1080/15228830802094627.CrossRefGoogle Scholar
- Asselbergs, J., Ruwaard, J., Ejdys, M., Schrader, N., Sijbrandij, M., & Riper, H. (2016). Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: An explorative study. Journal of Medical Internet Research, 18(3), e72. https://doi.org/10.2196/jmir.5505.CrossRefPubMedPubMedCentralGoogle Scholar
- Baggett, K. M., Davis, B., Feil, E. G., Sheeber, L. B., Landry, S. H., Carta, J. J., & Leve, C. (2010). Technologies for expanding the reach of evidence-based interventions: Preliminary results for promoting social-emotional development in early childhood. Topics in Early Childhood Special Education, 29(4), 226–238. https://doi.org/10.1177/0271121409354782.CrossRefPubMedPubMedCentralGoogle Scholar
- Bauer, M., Wylie, C., Jackson, T., Mislevy, B., Hoffman-John, E., John, M., & Corrigan, S. (2017). Why video games can be a good fit for formative assessment. Journal of Applied Testing Technology, 18(S1), 19–31Google Scholar
- Baumeister, R., Schmeichel, B., & Vohs, K. (2008). Self-regulation and the executive function: The self as controlling agent. In A. W. Kruglanski & E. T. Higgins (Eds.), Social psychology: Handbook of basic principles (2nd edn., p. 516). New York: The Guilford Press.Google Scholar
- Bessell, T. L., McDonald, S., Silagy, C. A., Anderson, J. N., Hiller, J. E., & Sansom, L. N. (2002). Do Internet interventions for consumers cause more harm than good? A systematic review. Health Expectations, 5(1), 28–37. https://doi.org/10.1046/j.1369-6513.2002.00156.x.CrossRefPubMedPubMedCentralGoogle Scholar
- Boonstra, T. W., Nicholas, J., Wong, Q. J., Shaw, F., Townsend, S., & Christensen, H. (2018). Using mobile phone sensor technology for mental health research: Integrated analysis to identify hidden challenges and potential solutions. Journal of Medical Internet Research, 20(7), e10131. https://doi.org/10.2196/10131.CrossRefPubMedPubMedCentralGoogle Scholar
- Burns, J. M., Davenport, T. A., & Christensen, H. (2013). Game on: Exploring the impact of technologies on young men’s mental health and wellbeing. Findings from the first Young and Well National Survey. Retrieved from http://scholar.google.com/scholar?hl=en%5C&btnG=Search%5C&q=intitle:Game+On:+Exploring+the+Impact+of+Technologies+on+Young+Men’s+Mental+Health+and+Wellbeing%5C#0.Google Scholar
- Campbell, S. B., Denham, S. A., Howarth, G. Z., Jones, S. M., Whittaker, J. V., Williford, A. P., … Darling-Churchill, K. (2016). Commentary on the review of measures of early childhood social and emotional development: Conceptualization, critique, and recommendations. Journal of Applied Developmental Psychology, 45, 19–41. https://doi.org/10.1016/j.appdev.2016.01.008.CrossRefGoogle Scholar
- Carlson, S. M., & Zelazo, P. D. (2014). Minnesota executive function scale: test manual. Reflection Sciences, LLC: St. Paul, MN.Google Scholar
- Casey, L. M., Oei, T. P. S., Raylu, N., Horrigan, K., Day, J., Ireland, M., & Clough, B. A. (2017). Internet-based delivery of cognitive behaviour therapy compared to monitoring, feedback and support for problem gambling: A randomised controlled trial. Journal of Gambling Studies. https://doi.org/10.1007/s10899-016-9666-y.CrossRefPubMedGoogle Scholar
- Cheng, Q., Li, T. M., Kwok, C.-L., Zhu, T., & Yip, P. S. (2017). Assessing suicide risk and emotional distress in chinese social media: A text mining and machine learning study. Journal of Medical Internet Research, 19(7), e243. https://doi.org/10.2196/jmir.7276.CrossRefPubMedPubMedCentralGoogle Scholar
- Clarke, J., Proudfoot, J., Birch, M.-R., Whitton, A. E., Parker, G., Manicavasagar, V., … Hadzi-Pavlovic, D. (2014). Effects of mental health self-efficacy on outcomes of a mobile phone and web intervention for mild-to-moderate depression, anxiety and stress: Secondary analysis of a randomised controlled trial. BMC Psychiatry, 14(1), 272. https://doi.org/10.1186/s12888-014-0272-1.CrossRefPubMedPubMedCentralGoogle Scholar
- Cummins, R. A., & Lau, A. L. D. (2005). Personal Wellbeing Index–School Children.Google Scholar
- Davidson, M. C., Amso, D., Anderson, L. C., & Diamond, A. (2006). Development of cognitive control and executive functions from 4 to 13 years: Evidence from manipulations of memory, inhibition, and task switching. Neuropsychologia, 44(11), 2037–2078. https://doi.org/10.1016/j.neuropsychologia.2006.02.006.CrossRefPubMedPubMedCentralGoogle Scholar
- Day, J. J., & Sanders, M. R. (2018). Do parents benefit from help when completing a self-guided parenting program online? A randomized controlled trial comparing triple p online with and without telephone support. Behavior Therapy. https://doi.org/10.1016/j.beth.2018.03.002.CrossRefPubMedGoogle Scholar
- Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/10.1146/annurev-psych-113011-143750.CrossRefGoogle Scholar
- DiCerbo, K. E., Shute, V., & Kim, Y. J. (2017). The future of assessment in technology-rich environments: Psychometric considerations. In J. M. Spector, B. Lockee, & M. Childress (Eds.), Learning, design, and technology: An international compendium of theory, research, practice, and policy (pp. 1–21). New York: Springer.Google Scholar
- Doherty, G., Coyle, D., & Sharry, J. (2012). Engagement with online mental health interventions: An Exploratory Clinical Study of a Treatment for Depression. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI’12 (p. 1421). New York: ACM Press. https://doi.org/10.1145/2207676.2208602.
- Dunton, G. F., Dzubur, E., & Intille, S. (2016). Feasibility and performance test of a real-time sensor-informed context-sensitive ecological momentary assessment to capture physical activity. Journal of Medical Internet Research, 18(6), e106. https://doi.org/10.2196/jmir.5398.CrossRefPubMedPubMedCentralGoogle Scholar
- Elliott, S. N., Davies, M. D., Frey, J. R., Gresham, F., & Cooper, G. (2018). Development and initial validation of a social emotional learning assessment for universal screening. Journal of Applied Developmental Psychology, 55, 39–51. https://doi.org/10.1016/j.appdev.2017.06.002.CrossRefGoogle Scholar
- Eyberg, S. M., & Pincus, D. (1999). Eyberg child behavior inventory and Sutter–Eyberg student behavior inventory—Revised: Professional manual. Odessa: Psychological Assessment Resources.Google Scholar
- Freeman, D., Reeve, S., Robinson, A., Ehlers, A., Clark, D., Spanlang, B., & Slater, M. (2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychological Medicine, 47(14), 2393–2400. https://doi.org/10.1017/S003329171700040X.CrossRefPubMedPubMedCentralGoogle Scholar
- Freiberg, K., & Homel, R. (2016). Rumble’s quest: A valid and reliable measure of social and emotional wellbeing for middle childhood. Retrieved December 13, 2018, from http://www.realwell.org.au/wp-content/uploads/2016/07/RQ-Psychometrics-Summary.pdf.
- Freiberg, K., Homel, R., & Lamb, C. (2007). The pervasive impact of poverty on children: Tackling family adversity and promoting child development through the pathways to prevention project. In A. France & R. Homel (Eds.), Pathways and crime prevention: Theory, policy and practice (Vol 226, pp. 226–246). Cullumpton: Willan Publishing.Google Scholar
- Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2015). BRIEF-2: Behavior rating inventory of executive function. Lutz, FL: Psychological Assessment Resources.Google Scholar
- Goodenow, C. (1993). The psychological sense of school membership among adolescents: Scale development and educational correlates. Psychology in the Schools, 30(1), 79–90. https://doi.org/10.1002/1520-6807(199301)30:1%3C79::AID-PITS2310300113>3.0.CO;2-XCrossRefGoogle Scholar
- Harter, S. (1985). Manual for the self-perception profile for children. Denver: University of Denver.Google Scholar
- Heaton, R. K., Chelune, G. J., Talley, J. L., Kay, G. G., & Curtiss, G. (1993). Wisconsin Card Sorting Test (WCST): Manual: Revised and Expanded. Odessa, FL: Psychological Assessment Resources (PAR).Google Scholar
- Jones, S. M., Bailey, R., Barnes, S. P., & Partee, A. (2016). Executive Function Mapping Project: Untangling the Terms and Skills Related to Executive Function and Self-Regulation in Early Childhood. OPRE Report #2016-88. Washington, DC. Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/efmapping_report_101416_final_508.pdf.
- Kagan, J. (1964). Matching familiar figures test. Cambridge: Harvard University.Google Scholar
- Kelders, S. M., Sommers-Spijkerman, M., & Goldberg, J. (2018). Investigating the direct impact of a gamified versus nongamified well-being intervention: an exploratory experiment. Journal of Medical Internet Research, 20(7), e247. https://doi.org/10.2196/jmir.9923.CrossRefPubMedPubMedCentralGoogle Scholar
- Kochanska, G., Murray, K., Jacques, T. Y., Koenig, A. L., & Vandegeest, K. A. (1996). Inhibitory control in young children and Its role in emerging internalization. Child Development, 67(2), 490–507. https://doi.org/10.1111/j.1467-8624.1996.tb01747.x.CrossRefPubMedGoogle Scholar
- Kramer, J., Velden, M. T., Kafkes, A., Basu, S., Federico, J., & Kielhofner, G. (2014). Child occupational self assessment. Chicago: Model of Human Occupation Clearinghouse.Google Scholar
- Liu, S., Zhu, M., Yu, D. J., Rasin, A., & Young, S. D. (2017). Using real-time social media technologies to monitor levels of perceived stress and emotional state in college students: a web-based questionnaire study. JMIR Mental Health, 4(1), e2. https://doi.org/10.2196/mental.5626.CrossRefPubMedPubMedCentralGoogle Scholar
- Luciana, M., & Nelson, C. A. (2002). Assessment of neuropsychological function through use of the cambridge neuropsychological testing automated battery: Performance in 4- to 12-year-old children. Developmental Neuropsychology, 22(3), 595–624. https://doi.org/10.1207/S15326942DN2203_3.CrossRefPubMedGoogle Scholar
- Manly, T., Robertson, I. H., Anderson, V., & Nimmo-Smith, I. (1999). TEA-Ch: The test of everyday attention for children manual. Bury St. Edmunds: Thames Valley Test Company Limited.Google Scholar
- Merry, S. N., Stasiak, K., Shepherd, M., Frampton, C., Fleming, T., & Lucassen, M. F. G. (2012). The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: Randomised controlled non-inferiority trial. BMJ, 344, e2598–e2598. https://doi.org/10.1136/bmj.e2598.CrossRefPubMedPubMedCentralGoogle Scholar
- Mislevy, R. J., Oranje, A., Bauer, M. I., von Davier, A., Hao, J., Corrigan, S., … John, M. (2014). Psychometric considerations in game-based assessment. GlassLab Report. Retrieved from https://web.archive.org/web/20160320151604/http://www.instituteofplay.org/wp-content/uploads/2014/02/GlassLab_GBA1_WhitePaperFull.pdf.
- Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “Frontal Lobe” Tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49–100. https://doi.org/10.1006/cogp.1999.0734.CrossRefPubMedGoogle Scholar
- Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., Harrington, H., … Caspi, A. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences of the United States of America, 108(7), 2693–2698. https://doi.org/10.1073/pnas.1010076108.CrossRefPubMedPubMedCentralGoogle Scholar
- Mueller, S. T. (2018). The Psychology Experiment Building Language. Retrieved from http://pebl.sourceforge.net/.
- Neubauer, K., von Auer, M., Murray, E., Petermann, F., Helbig-Lang, S., & Gerlach, A. L. (2013). Internet-delivered attention modification training as a treatment for social phobia: A randomized controlled trial. Behaviour Research and Therapy, 51(2), 87–97. https://doi.org/10.1016/j.brat.2012.10.006.CrossRefPubMedPubMedCentralGoogle Scholar
- Newby, J. M., Mackenzie, A., Williams, A. D., McIntyre, K., Watts, S., Wong, N., & Andrews, G. (2013). Internet cognitive behavioural therapy for mixed anxiety and depression: a randomized controlled trial and evidence of effectiveness in primary care. Psychological Medicine, 43(12), 2635–2648. https://doi.org/10.1017/S0033291713000111.CrossRefPubMedGoogle Scholar
- Nigg, J. T. (2017). Annual Research Review: On the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychopathology. Journal of Child Psychology and Psychiatry, 58(4), 361–383. https://doi.org/10.1111/jcpp.12675.CrossRefPubMedGoogle Scholar
- Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., … Riley, W. T. (2016). The pace of technologic change: Implications for digital health behavior intervention research. American Journal of Preventive Medicine, 51(5), 816–824. https://doi.org/10.1016/j.amepre.2016.05.001.CrossRefPubMedGoogle Scholar
- Pennant, M. E., Loucas, C. E., Whittington, C., Creswell, C., Fonagy, P., Fuggle, P., … Kendall, T. (2015). Computerised therapies for anxiety and depression in children and young people: A systematic review and meta-analysis. Behaviour Research and Therapy, 67, 1–18. https://doi.org/10.1016/j.brat.2015.01.009.CrossRefPubMedGoogle Scholar
- Picard, L., Abram, M., Orriols, E., & Piolino, P. (2017). Virtual reality as an ecologically valid tool for assessing multifaceted episodic memory in children and adolescents. International Journal of Behavioral Development, 41(2), 211–219. https://doi.org/10.1177/0165025415616198.CrossRefGoogle Scholar
- Ponitz, C. E. C., McClelland, M. M., Jewkes, A. M., Connor, C. M., Farris, C. L., & Morrison, F. J. (2008). Touch your toes! developing a direct measure of behavioral regulation in early childhood. Early Childhood Research Quarterly, 23(2), 141–158. https://doi.org/10.1016/j.ecresq.2007.01.004.CrossRefGoogle Scholar
- Quiroz, J. C., Geangu, E., & Yong, M. H. (2018). Emotion-recognition using smart watch sensor data: Mixed-design study (Preprint). JMIR Mental Health. https://doi.org/10.2196/10153.
- Raines, T. C., Dever, B. V., Kamphaus, R. W., & Roach, A. T. (2012). Universal screening for behavioral and emotional risk: A promising method for reducing disproportionate placement in special education. The Journal of Negro Education, 81(3), 283–296. https://doi.org/10.7709/jnegroeducation.81.3.0283.CrossRefGoogle Scholar
- Riva, G. (1997). Virtual reality in neuro-psycho-physiology: Cognitive, clinical and methodological issues in assessment and rehabilitation. Amsterdam: IOS Press.Google Scholar
- Sandercock, G. R. H., Alibrahim, M., & Bellamy, M. (2016). Media device ownership and media use: Associations with sedentary time, physical activity and fitness in English youth. Preventive Medicine Reports, 4, 162–168. https://doi.org/10.1016/j.pmedr.2016.05.013.CrossRefPubMedPubMedCentralGoogle Scholar
- Schlegl, S., Bürger, C., Schmidt, L., Herbst, N., & Voderholzer, U. (2015). The potential of technology-based psychological interventions for anorexia and bulimia nervosa: A systematic review and recommendations for future research. Journal of Medical Internet Research, 17(3), e85. https://doi.org/10.2196/jmir.3554.CrossRefPubMedPubMedCentralGoogle Scholar
- Shapiro, E. S. (2000). School psychology from an instructional perspective: Solving big, not little problems. School Psychology Review, 29(4), 560.Google Scholar
- Shute, V. J., Ke, F., & Wang, L. (2017). Assessment and Adaptation in Games. In P. Wouters & H. van Oostendorp (Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp. 59–78). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-39298-1_4.CrossRefGoogle Scholar
- Somech, L. Y., & Elizur, Y. (2012). Promoting self-regulation and cooperation in pre-kindergarten children with conduct problems: A randomized controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 51(4), 412–422. https://doi.org/10.1016/j.jaac.2012.01.019.CrossRefPubMedGoogle Scholar
- Spence, S. H., Donovan, C. L., March, S., Gamble, A., Anderson, R., Prosser, S., … Kenardy, J. (2008). Online CBT in the treatment of child and adolescent anxiety disorders: Issues in the development of BRAVE–ONLINE and two case illustrations. Behavioural and Cognitive Psychotherapy, 36(04), 411–430. https://doi.org/10.1017/S135246580800444X.CrossRefGoogle Scholar
- Spence, S. H., Donovan, C. L., March, S., Gamble, A., Anderson, R. E., Prosser, S., & Kenardy, J. (2011). A randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. Journal of Consulting and Clinical Psychology, 79(5), 629–642. https://doi.org/10.1037/a0024512.CrossRefPubMedGoogle Scholar
- Strecher, V. J. (2007). Internet methods for delivering behavioral and health-related interventions (eHealth). Annual Review of Clinical Psychology, 3(1), 53–76. https://doi.org/10.1146/annurev.clinpsy.3.022806.091428.CrossRefPubMedGoogle Scholar
- Tsukayama, E., Toomey, S. L., Faith, M. S., & Duckworth, A. L. (2010). Self-control as a protective factor against overweight status in the transition from childhood to adolescence. Archives of Pediatrics & Adolescent Medicine, 164(7). https://doi.org/10.1001/archpediatrics.2010.97.
- Turner, K. M. T., Baker, S., & Day, J. J. (2018). Technology-assisted delivery of parenting programs. In T. G. Mazzucchelli & M. R. Sanders (Eds.), The power of positive parenting: Transforming the lives of children, parents and communities using the Triple P system (1st ed.). Oxford: Oxford University Press.Google Scholar
- Wagner, B., Fitzpatrick, J. P., Mazzucchelli, T. G., Symons, M., Carmichael Olson, H., Jirikowic, T., … Latimer, J. (2018). Study protocol for a self-controlled cluster randomised trial of the Alert Program to improve self-regulation and executive function in Australian Aboriginal children with fetal alcohol spectrum disorder. BMJ Open, 8(3), e021462.CrossRefPubMedPubMedCentralGoogle Scholar
- Weintraub, S., Bauer, P. J., Zelazo, P. D., Wallner-Allen, K., Dikmen, S. S., Heaton, R. K., … Gershon, R. C. (2013). I. NIH Toolbox cognition battery (CB): Introduction and pediatric data. Monographs of the Society for Research in Child Development, 78(4), 1–15. https://doi.org/10.1111/mono.12031.CrossRefPubMedPubMedCentralGoogle Scholar
- Yamamoto, N., & Imai-Matsumura, K. (2017). Gender differences in executive function and behavioural self-regulation in 5 years old kindergarteners from East Japan. Early Child Development and Care, 1–12. https://doi.org/10.1080/03004430.2017.1299148.
- Yardley, L., Spring, B. J., Riper, H., Morrison, L. G., Crane, D. H., Curtis, K., … Blandford, A. (2016). Understanding and promoting effective engagement with digital behavior change interventions. American Journal of Preventive Medicine, 51(5), 833–842. https://doi.org/10.1016/j.amepre.2016.06.015.CrossRefPubMedGoogle Scholar
- Zapata-Rivera, D., & Bauer, M. (2012). Exploring the role of games in educational assessment. Technology-based assessments for twenty-first-century skills: Theoretical and practical implications from modern Research, 147–169.Google Scholar
- Zhou, Q., Chen, S. H., & Main, A. (2012). Commonalities and differences in the research on children’s effortful control and executive function: A call for an integrated model of self-regulation. Child Development Perspectives, 6(2), 112–121. https://doi.org/10.1111/j.1750-8606.2011.00176.x.CrossRefGoogle Scholar