Internet-Based, Unproctored Assessments on Mobile and Non-Mobile Devices: Usage, Measurement Equivalence, and Outcomes
- 563 Downloads
The purpose of this study was to determine the usage rates, measurement equivalence, and potential outcome differences between mobile and non-mobile device-based deliveries of an unproctored, non-cognitive assessment.
This study utilized a quasi-experimental design based on archival data obtained from applicants who completed a non-cognitive assessment on a mobile (n = 7,743; e.g., smartphones, tablet computers) or non-mobile (n = 929,341; e.g., desktop computers) device as part of an operational, high-stakes pre-employment selection process.
One percent of applicants used mobile devices to complete the assessment. Multiple-group confirmatory factor analysis indicated the assessment was equivalent across mobile and non-mobile devices at the configural, metric, scalar, and latent mean levels. A comparison of observed score means using one-way and factorial ANOVAs demonstrated that the use of mobile and non-mobile devices did not produce any practically significant score differences on the assessment across devices or applicant demographic subgroups.
Industry and technological trends suggest mobile device usage will only increase. Thus, demonstrating that mobile device functionality and hardware characteristics do not change the psychometric functioning or applicant outcomes for a non-cognitive, text-based selection assessment is critical to talent assessment.
This study provides the first empirical examination of the usage of mobile devices to complete talent assessments and their impact on assessment properties and applicant outcomes, and serves as the foundation for future research and application of this growing technological trend in pre-employment assessment.
KeywordsMeasurement equivalence Measurement invariance Mobile devices Selection and assessment Talent assessment and selection Internet-based testing Unproctored testing Technology-enhanced assessment Non-cognitive assessment
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: AuthorsGoogle Scholar
- American Psychological Association. (1986). Guidelines for computer-based tests and interpretations. Washington, DC: Author.Google Scholar
- American Psychological Association. (2010). Ethical principles of psychologists and code of conduct. Washington, DC: Author.Google Scholar
- Bartram, D. (2006). Testing on the internet: Issues, challenges and opportunities in the field of occupational assessment. In D. Bartram & R. K. Hambleton (Eds.), Computer-based testing and the internet: Issues and advances (pp. 13–37). San Francisco, CA: Wiley.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Doverspike, D., Arthur Jr., W., Taylor, J., & Carr, A. (2012, April). Mobile mania: The impact of device type on remotely delivered assessments. In J. Scott (chair), Chasing the tortoise: Zeno’s paradox in technology-based assessment. Paper presented at the 27th annual conference of The Society for Industrial and Organizational Psychology, San Diego, CA. Google Scholar
- Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor & Department of Justice. (1978). Uniform guidelines on employee selection procedures. Washington, DC: Authors.Google Scholar
- Fallaw, S. S., & Kantrowitz, T. M. (2011). 2011 Global assessment trends report [White paper]. Retrieved from http://www2.Shl.com/campaign/2011-global-assessment-trends-report/thankyou.aspx.
- Fallaw, S. S., Kantrowitz, T. M., & Dawson, C. R. (2012). 2012 Global assessment trends report [White paper]. Retrieved from http://www.Shl.com/assets/GATR_2012_US.pdf.
- Hough, L. M. (2010). Assessment of background and life experience: The past as prologue. In J. Scott & J. Waclawski (Eds.), Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent (pp. 109–140). San Francisco, CA: Wiley.Google Scholar
- Huff, K. C. (2006). The effects of mode of administration on timed cognitive ability tests. Unpublished doctoral dissertation, North Carolina State University, Raleigh, NCGoogle Scholar
- International Test Commission. (2005). International guidelines on computer-based and Internet delivered testing. Granada, Spain: Author.Google Scholar
- Johnson, J. J. (2000, March). Web-based personality assessment project description and rationale. Poster presented at the annual meeting of the Eastern Psychological Association, Baltimore, MD.Google Scholar
- Jourbert, T., & Kriek, H. J. (2009). Psychometric comparison of paper-and-pencil and online personality assessments in a selection setting. South African Journal of Industrial Psychology, 35(1), 78–88.Google Scholar
- Muthén, L. K., & Muthén, B. O. (2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
- NPD Group. (2013, February). U.S. consumer technology retail sales decline 2 percent in 2012. NPD Group Press Release. Retrieved from https://www.npd.com/wps/portal/npd/us/news/press-releases/us-consumer-technology-retail-sales-decline-2-percent-in-2012-according-to-the-npd-group/.
- Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.Google Scholar
- Reynolds, D. H., & Rupp, D. E. (2010). Advances in technology-facilitated assessment. In J. C. Scott & D. H. Reynolds (Eds.), Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent (pp. 609–641). San Francisco, CA: Jossey-Bass.Google Scholar
- Scott, J. C., & Mead, A. D. (2011). Foundations for measurement. In N. Tippins & S. Adler (Eds.), Technology-enhanced assessment of talent (pp. 1–18). San Francisco, CA: Wiley.Google Scholar
- Smith, A. (2012, March 1). 46% of American adults are smartphone owners. Retrieved from http://PewInternet.org/Reports/2012/Smartphone-Update-2012.aspx.
- Society for Industrial & Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). Bowling Green, OH: Author.Google Scholar
- Stevens, J. (1990). Intermediate statistics: A modern approach. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Waters, S. D., & Pommerich, M. (2007, April). Context effects in internet testing: A literature review. Paper presented at the 22nd annual conference of The Society for Industrial and Organizational Psychology, New York City, NY.Google Scholar