Advertisement

MultiSCED: A tool for (meta-)analyzing single-case experimental data with multilevel modeling

  • Lies DeclercqEmail author
  • Wilfried Cools
  • S. Natasha Beretvas
  • Mariola Moeyaert
  • John M. Ferron
  • Wim Van den Noortgate
Article

Abstract

The MultiSCED web application has been developed to assist applied researchers in behavioral sciences to apply multilevel modeling to quantitatively summarize single-case experimental design (SCED) studies through a user-friendly point-and-click interface embedded within R. In this paper, we offer a brief introduction to the application, explaining how to define and estimate the relevant multilevel models and how to interpret the results numerically and graphically. The use of the application is illustrated through a re-analysis of an existing meta-analytic dataset. By guiding applied researchers through MultiSCED, we aim to make use of the multilevel modeling technique for combining SCED data across cases and across studies more comprehensible and accessible.

Keywords

Single-case experimental design SCED Multilevel analysis Shiny 

Notes

References

  1. Auerbach, C., & Schudrich, W. Z. (2013). SSD for R: A comprehensive statistical package to analyze single-system data. Research on Social Work Practice, 23(3), 346–353.  https://doi.org/10.1177/1049731513477213 CrossRefGoogle Scholar
  2. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412.  https://doi.org/10.1016/j.jml.2007.12.005 CrossRefGoogle Scholar
  3. Baek, E. K., & Ferron, J. M. (2013). Multilevel models for multiple-baseline data: Modeling across-participant variation in autocorrelation and residual variance. Behavior Research Methods, 45(1), 65–74.  https://doi.org/10.3758/s13428-012-0231-z CrossRefGoogle Scholar
  4. Baek, E. K., et al. (2016). Using visual analysis to evaluate and refine multilevel models of single-case studies. Journal of Special Education, 50(1), 18–26.  https://doi.org/10.1177/0022466914565367 CrossRefGoogle Scholar
  5. Bates, D. (2006). lmer, p-values and all that. https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html
  6. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.  https://doi.org/10.18637/jss.v067.i01.CrossRefGoogle Scholar
  7. Bolker, B., et al. (2018). GLMM FAQ. http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html
  8. Bulté, I., & Onghena, P. (2013). The single-case data analysis package: Analysing single-case experiments with R software. Journal of Modern Applied Statistical Methods, 12(2), 450–478.  https://doi.org/10.1017/CBO9781107415324.004 CrossRefGoogle Scholar
  9. Chang, W., et al. (2017). Shiny: Web application framework for R. https://cran.r-project.org/package=shiny
  10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale: Erlbaum.Google Scholar
  11. Declercq, L., et al. (2018). Analysis of single-case experimental count data using the linear mixed effects model: A simulation study. Behavior Research Methods.  https://doi.org/10.3758/s13428-018-1091-y
  12. Dibley, S., & Lim, L. (1999). Providing choice making opportunities within and between daily school routines. Journal of Behavioral Education, 9 (2), 117–132.  https://doi.org/10.1023/A:1022888917128 CrossRefGoogle Scholar
  13. Ferron, J. M., et al. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41(2), 372–384.  https://doi.org/10.3758/BRM.41.2.372 CrossRefGoogle Scholar
  14. Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42(4), 930–943.  https://doi.org/10.3758/BRM.42.4.930 CrossRefGoogle Scholar
  15. Fisch, G. S. (2001). Evaluating data from behavioral analysis: Visual inspection or statistical models? Behavioural Processes, 54(1-3), 137–154.  https://doi.org/10.1016/S0376-6357(01)00155-3 CrossRefGoogle Scholar
  16. Jamshidi, L., et al. (2018). Review of single-subject experimental design meta-analyses and reviews: 1985-2015. Submitted for publication.Google Scholar
  17. Joo, S. H., et al. (2017). Approaches for specifying the level-1 error structure when synthesizing single-case data. Journal of Experimental Education, 0973, 1–20.  https://doi.org/10.1080/00220973.2017.1409181 CrossRefGoogle Scholar
  18. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3), 983–997.  https://doi.org/10.2307/2533558 CrossRefGoogle Scholar
  19. Kern, L., et al. (2001). Choice of task sequence to reduce problem behaviors. Journal of Positive Behavior Interventions, 3(1), 3– 10.  https://doi.org/10.1177/109830070100300102. http://search.ebscohost.com/login.aspx?direct=true%7B%5C&%7Ddb=psyh%7B%5C&%7DAN=2001-14259-001%7B%5C&%7Dsite=ehostlive CrossRefGoogle Scholar
  20. Littell, R. C., et al. (2007). SAS for mixed models. 2nd ed., 2. Cary: SAS Institute.Google Scholar
  21. Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502.  https://doi.org/10.3758/s13428-016-0809-y CrossRefGoogle Scholar
  22. Manolov, R., & Solanas, A. (2013). A comparison of mean phase difference and generalized least squares for analyzing single-case data. Journal of School Psychology, 51(2), 201–215.  https://doi.org/10.1016/j.jsp.2012.12.005 CrossRefGoogle Scholar
  23. Manolov, R., & Rochat, L. (2015). Further developments in summarising and meta-analysing single-case data: An illustration with neurobehavioural interventions in acquired brain injury. Neuropsychological Rehabilitation, 25 (5), 637–662.  https://doi.org/10.1080/09602011.2015.1064452 CrossRefGoogle Scholar
  24. Manolov, R., & Moeyaert, M. (2017). How can single-case data be analyzed? Software resources, tutorial, and reflections on analysis. Behavior Modification, 41(2), 179–228.  https://doi.org/10.1177/0145445516664307 CrossRefGoogle Scholar
  25. Manolov, R., & Solanas, A. (2018). Quantifying differences between conditions in single-case designs: Possible analysis and meta-analysis. Developmental Neurorehabilitation, 21(4), 238–252.  https://doi.org/10.3109/17518423.2015.1100688 CrossRefGoogle Scholar
  26. Moes, D. R. (1998). Integrating choice-making opportunities within teacher-assigned academic tasks to facilitate the performance of children with autism. Journal of the Association for Persons with Severe Handicaps, 23(4), 319–328.CrossRefGoogle Scholar
  27. Moeyaert, M., et al. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48(5), 719–748.  https://doi.org/10.1080/00273171.2013.816621 CrossRefGoogle Scholar
  28. Moeyaert, M., et al. (2014). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52(2), 191–211.  https://doi.org/10.1016/j.jsp.2013.11.003 CrossRefGoogle Scholar
  29. Moeyaert, M., et al. (2015). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly, 30(1), 50–63.  https://doi.org/10.1037/spq0000068 CrossRefGoogle Scholar
  30. Moeyaert, M., et al. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods, 22(4), 760–778.  https://doi.org/10.1037/met0000136 CrossRefGoogle Scholar
  31. Nagler, E., Rindskopf, D. M., & Shadish, W. R. (2008). Analyzing data from small N designs using multilevel models: A procedural handbook. Unpublished manuscript.Google Scholar
  32. Onghena, P., & Edgington, E. S. (2005). Customization of pain treatments: Single-case design and analysis. The Clinical Journal of Pain, 21(1), 56–68.  https://doi.org/10.1097/00002508-200501000-00007 CrossRefGoogle Scholar
  33. Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44(3), 795–805.  https://doi.org/10.3758/s13428-011-0180-y CrossRefGoogle Scholar
  34. Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T.R. Kratochwill, J.R. Levin, & N. J. Hillsdale (Eds.) Single-case research design and analysis: New directions for psychology and education. Erlbaum. Chap. 2 (pp. 15–38).Google Scholar
  35. Peugh, J. L., & Enders, C. K. (2005). Using the SPSS mixed procedure to fit cross-sectional and longitudinal multilevel models. Educational and Psychological Measurement, 65(5), 717–741.  https://doi.org/10.1177/0013164405278558 CrossRefGoogle Scholar
  36. Pinheiro, J., et al. (2018). nlme: Linear and nonlinear mixed effects models. R package version 3. pp. 1-137. https://CRAN.R-project.org/package=nlme
  37. Pustejovsky, J. E. (2016) scdhlm: Estimating hierarchical linear models for single-case designs. R package version 0.3. Austin: University of Texas. http://github.com/jepusto/scdhlm Google Scholar
  38. R Core Team (2013). R: A language and environment for statistical computing. Vienna, Austria. http://www.r-project.org/
  39. Rasbash, J., et al. (2009). A user’s guide to MLwiN, Version 2.10, pp. 1–296. http://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/mlwin-userman-09.pdf
  40. Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2013). HLM 7.01 for Windows [computer software]. Skokie, IL.Google Scholar
  41. Rindskopf, D. M., & Ferron, J. M. (2014). Using multilevel models to analyze single-case design data. In R.T. Kratochwill, & J.R. Levin (Eds.) Single-case intervention research: Methodological and statistical advances. School psychology series. Washington, DC, US: American Psychological Association, pp. 221–246.  https://doi.org/10.1037/14376-008
  42. Rodabaugh, E., & Moeyaert, M. (2017). Multilevel modeling of single-case data: An introduction and tutorial for the applied researcher. In NERA Conference Proceedings 2017 8.Google Scholar
  43. Romaniuk, C., et al. (2002). The influence of activity choice on problem behaviors maintained by escape versus attention. Journal of Applied Behavior Analysis, 35(4), 349–362.  https://doi.org/10.1901/jaba.2002.35-349. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1284398/ CrossRefGoogle Scholar
  44. Shadish, W. R. (2014). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23(2), 139–146.  https://doi.org/10.1177/0963721414524773 CrossRefGoogle Scholar
  45. Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43(4), 971–980.  https://doi.org/10.3758/s13428-011-0111-y CrossRefGoogle Scholar
  46. Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 188–196. https://doi.org/10.1080/17489530802581603. http://search.ebscohost.com/login.aspx?direct=true%7B%5C&%7Dlang=zh-tw%7B%5C&%7Dsite=ehost-live CrossRefGoogle Scholar
  47. Shadish, W. R., Kyse, E. N., & Rindskopf, D. M. (2013). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18(3), 385–405.  https://doi.org/10.1037/a0032964 CrossRefGoogle Scholar
  48. Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choice-making as an intervention for problem behavior: A meta-analysis. Journal of Positive Behavior Interventions, 6(4), 228–237.  https://doi.org/10.1177/10983007040060040401 CrossRefGoogle Scholar
  49. Ugille, M., et al. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44(4), 1244–1254.  https://doi.org/10.3758/s13428-012-0213-1 CrossRefGoogle Scholar
  50. Valentine, J. C., et al. (2016). Between-case standardized mean difference effect sizes for single-case designs: A primer and tutorial using the scdhlm web application. Oslo, Norway: The Campbell Collaboration.  https://doi.org/10.4073/cmdp.2016.1. https://campbellcollaboration.org
  51. Van den Noortgate, W., et al. (2014). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47(4), 1274–1294.  https://doi.org/10.3758/s13428-014-0527-2 CrossRefGoogle Scholar
  52. Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18(3), 325–346.  https://doi.org/10.1521/scpq.18.3.325.22577 CrossRefGoogle Scholar
  53. Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, and Computers, 35(1), 1–10.  https://doi.org/10.3758/BF03195492 CrossRefGoogle Scholar
  54. Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention, 2 (3), 142–151.  https://doi.org/10.1080/17489530802505362 CrossRefGoogle Scholar
  55. Welham, S. J., & Thompson, R. (1997). Likelihood ratio tests for fixed model terms using residual maximum likelihood. Journal of the Royal Statistical Society, Series B: Statistical Methodology, 59(3), 701–714.  https://doi.org/10.2307/2346019 CrossRefGoogle Scholar
  56. Wickham, H. (2009) ggplot2: Elegant graphics for data analysis. New York: Springer. ISBN: 978-0-387-98140-6. http://ggplot2.org.CrossRefGoogle Scholar
  57. Wickham, H. (2011). The split-apply-combine strategy for data analysis. Journal of Statistical Software, 40 (1), 1–29. http://www.jstatsoft.org/v40/i01/ CrossRefGoogle Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  • Lies Declercq
    • 1
    Email author
  • Wilfried Cools
    • 1
  • S. Natasha Beretvas
    • 2
  • Mariola Moeyaert
    • 3
  • John M. Ferron
    • 4
  • Wim Van den Noortgate
    • 1
  1. 1.Faculty of Psychology and Educational Sciencesimec - ITEC, KU LeuvenLeuvenBelgium
  2. 2.Department of Educational PsychologyUniversity of TexasAustinUSA
  3. 3.Department of Educational Psychology and MethodologyUniversity at Albany-State University of New YorkAlbanyUSA
  4. 4.Department of Educational Measurement and ResearchUniversity of South FloridaTampaUSA

Personalised recommendations