A Multiple-Indicator Latent Growth Mixture Model to Track Courses with Low-Quality Teaching

Abstract

This paper describes a multi-indicator latent growth mixture model built on the data collected by a large Italian university to track students’ satisfaction over time. The analysis of the data involves two steps: first, a pre-processing of data selects the items to be part of the synthetic indicator that measures students’ satisfaction; the second step then retrieves heterogeneity that allows the identification of a clustering structure with a group of university courses (outliers) which underperform in terms of students’ satisfaction over time. Regression components of the model identify courses in need of further improvement and that are prone to receiving low classifications from students. Results show that it is possible to identify a large group of didactic activities with a high satisfaction level that stays constant over time; there is also a small group of problematic didactic activities with low satisfaction that decreases over the period under analysis.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    We use course and didactic activity as synonymous. To be more precise, didactic activity is understood as the course-teacher pair.

  2. 2.

    Since we have longitudinal data for only three consecutive waves, only a linear latent trajectory may be specified.

  3. 3.

    Exploratory factor analysis on the data referring to the three academic years gave similar results; therefore, we decided to report only for the most recent academic year 2014–2015. Extracting factors with principal-components factoring gave the same solution; on the other hand, extracting the factors with methods such as iterative maximum likelihood did not provide good results as a Heywood case was encountered (Fabrigar et al. 1999). KMO index is equal to 0.956.

  4. 4.

    Three-component mixture models always perform worse than the two-component mixture model (based on BIC).

  5. 5.

    The number of credits has been excluded from the potential covariates since it is highly correlated with the number of hours.

  6. 6.

    The classification is based on the estimates given by Eq. (7) and assigned to the latent class with maximum probability (Bayes’ optimal rule). The relative entropy index, which varies between 0 (min) and 1 (max), is 0.908. Thus, we have a good level of separation of the mixture components (Ramaswamy et al. 1993).

References

  1. Akaike, H. (1974). A new look at statistical-model identification. IEEE Transactions on Automatic Control, AC, 19(6), 716–723.

    Article  Google Scholar 

  2. Alves, B. C., & Dias, J. G. (2015). Survival mixture models in behavioral scoring. Expert Systems with Applications, 42(2), 3902–3910.

    Article  Google Scholar 

  3. Bacci, S., Bartolucci, F., Grilli, L., & Rampichini, C. (2017). Evaluation of student performance through a multidimensional finite mixture IRT model. Multivariate Behavioral Research, 52(6), 732–746.

    Article  Google Scholar 

  4. Baker, R. S. (2014). Educational data mining: An advance for intelligent systems in education. IEEE Intelligent Systems, 29(3), 78–82.

    Article  Google Scholar 

  5. Bartholomew, D. J., & Knott, M. (1999). Latent variable models and factor analysis. London: Arnold.

    Google Scholar 

  6. Bartolucci, F., Bacci, S., & Gnaldi, M. (2015). Statistical analysis of questionnaires: A unified approach based on R and stata. London: Chapman and Hall/CRC.

    Google Scholar 

  7. Bartolucci, F., Farcomeni, A., & Pennoni, F. (2012). Latent Markov models for longitudinal data. Boca Raton: Chapman & Hall/CRC.

    Google Scholar 

  8. Bassi, F. (2010). Experiential goods and customer satisfaction: An application to movies. Quality Technology & Quantitative Management, 7(1), 51–67.

    Article  Google Scholar 

  9. Bassi, F. (2016). Dynamic segmentation with growth mixture models. Advances in Data Analysis and Classification, 10(2), 263–279.

    Article  Google Scholar 

  10. Bassi, F., Clerici, R., & Aquario, D. (2017). Students’ evaluation of teaching at a large Italian university: measurement scale validation. Electronic Journal of Applied Statistical Analysis, 10(1), 93–117.

    Google Scholar 

  11. Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970). A maximization technique occurring in statistical analysis of probabilistic functions of Markov chains. Annals of Mathematical Statistics, 41(1), 164–171.

    Article  Google Scholar 

  12. Biemer, P. P. (2011). Latent class analysis of survey error. Hoboken: Wiley.

    Google Scholar 

  13. Bishop, J., Geiser, C., & Cole, D. A. (2015). Modeling latent growth with multiple indicators: A comparison of three approaches. Psychological Methods, 20(1), 43–62.

    Article  Google Scholar 

  14. Bollen, K. A., & Curran, P. J. (2006). Latent curve models: A structural equation approach. Hoboken: Wiley.

    Google Scholar 

  15. Boring, A., Ottoboni, K., & Stark, P. B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. Retrieved from Science Open.

  16. Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Economics of Education Review, 41(C), 71–88.

    Article  Google Scholar 

  17. Churchill, G. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 64–73.

    Article  Google Scholar 

  18. Clogg, C. C. (1995). Latent class models. In G. Arminger, C. C. Clogg, & M. E. Sobel (Eds.), Handbook of statistical modeling for the social and behavioral sciences (pp. 311–359). New York: Plenum Press.

    Google Scholar 

  19. Collins, L. M., & Lanza, S. T. (2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. Hoboken: Wiley.

    Google Scholar 

  20. Connel, A. M., & Frey, A. A. (2006). Growth mixture modelling in developmental psychology: overview and demonstration of heterogeneity in developmental trajectories of adolescent antisocial behaviour. Infant and Child Development, 15(6), 609–621.

    Article  Google Scholar 

  21. Dalla Zuanna, G., Bassi F., Clerici, R., Paccagnella, O., Paggiaro, A., Aquario D., Mazzuco C., Martinoia, S., Stocco, C., & Pierobon, S. (2015). Tools for teaching assessment at Padua University: role, development and validation. PRODID Project (Teacher professional development and academic educational innovation) (Report of Research Unit n.3). Padua: Department of Statistical Sciences, University of Padua.

  22. Dayton, C. M., & Macready, G. B. (1988). Concomitant-variable latent class models. Journal of the American Statistical Association, 83(401), 173–178.

    Article  Google Scholar 

  23. de Angelis, L., & Dias, J. G. (2014). Mining categorical sequences from data using a hybrid clustering method. European Journal of Operational Research, 234(3), 720–730.

    Article  Google Scholar 

  24. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society B, 39, 1–38.

    Google Scholar 

  25. Dias, J. G. (2006). Latent class analysis and model selection. In M. Spiliopoulou, R. Kruse, C. Borgelt, A. Nürnberger, & W. Gaul (Eds.), From data and information analysis to knowledge engineering (pp. 95–102). Berlin: Springer.

    Google Scholar 

  26. Dias, J. G. (2007). Model selection criteria for model-based clustering of categorical time series data: A Monte Carlo study. In R. Decker & H. J. Lenz (Eds.), Advances in data analysis (p. 23). Berlin: Springer.

    Google Scholar 

  27. Dias, J. G., & Ramos, S. B. (2014). The aftermath of the subprime crisis—a clustering analysis of world banking sector. Review of Quantitative Finance and Accounting, 42(2), 293–308.

    Article  Google Scholar 

  28. Dias, J. G., & Vermunt, J. K. (2007). Latent class modeling of website users’ search patterns: implications for online market segmentation. Journal of Retailing and Consumer Services, 14(6), 359–368.

    Article  Google Scholar 

  29. Dias, J. G., Vermunt, J. K., & Ramos, S. B. (2015). Clustering financial time series: New insights from an extended hidden Markov model. European Journal of Operational Research, 243(3), 852–864.

    Article  Google Scholar 

  30. Esling, P., & Agon, C. (2012). Time series data analysis. ACM Computing Survey, 45(19), 12.

    Google Scholar 

  31. European University Association. (2016). Quality culture in European Universities: A bottom up approach. Brussels: EUA.

    Google Scholar 

  32. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

    Article  Google Scholar 

  33. Formann, A. K. (1992). Linear logistic latent class analysis for polytomous data. Journal of the American Statistical Association, 87(418), 476–486.

    Article  Google Scholar 

  34. Frühwirth-Schnatter, S. (2006). Finite mixture and Markov switching models. New York: Springer.

    Google Scholar 

  35. Goodman, L. A. (1974). Exploratory latent factor analysis using both identifiable and unidentifiable models. Biometrika, 61(2), 215–231.

    Article  Google Scholar 

  36. Grilli, L., & Varriale, R. (2014). Specifying measurement error correlations in latent growth curve models with multiple indicators. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(4), 117–125.

    Article  Google Scholar 

  37. Grimm, K. J., & Widaman, K. F. (2010). Residual structures in latent growth curve modelling. Structural Equation Modeling, 17(3), 424–442.

    Article  Google Scholar 

  38. Hagenaars, J. A., & McCutcheon, A. L. (2002). Applied latent class analysis. Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  39. Harvey, L., & Green, D. (1993). Defining quality. Assessment and Evaluation in Higher Education, 18(1), 9–34.

    Article  Google Scholar 

  40. Hornstein, H. A. (2017). Student evaluations of teaching are an inadequate assessment tool for evaluating faculty performance. Cogent Education, 4(1), 1–8.

    Article  Google Scholar 

  41. La Rocca, M., Parrella, L., Primerano, I., Sulis, I., & Vitale, M. P. (2017). An integrated strategy for the analysis of student evaluation of teaching: from descriptive measures to explanatory models. Quality & Quantity, 51(2), 675–691.

    Article  Google Scholar 

  42. Lazarsfeld, P. F., & Henry, N. W. (1968). Latent structure analysis. Boston: Houghton Mifflin.

    Google Scholar 

  43. Masserini, L., Liberati, C., & Mariani, P. (2017). Quality service in banking: a longitudinal approach. Quality & Quantity, 51(2), 509–523.

    Article  Google Scholar 

  44. McArdle, J. J., & Epstein, D. (1987). Latent growth curves within developmental structural equation models. Child Development, 58, 110–133.

    Article  Google Scholar 

  45. McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York: Wiley.

    Google Scholar 

  46. Meggiolaro, S., Giraldo, A., & Clerici, R. (2017). A multilevel competing risks model for analysis of university students’ careers in Italy. Studies in Higher Education, 42(7), 1259–1274.

    Article  Google Scholar 

  47. Meredith, W., & Tisak, J. (1990). Latent curve analysis. Psychometrika, 55(1), 107–122.

    Article  Google Scholar 

  48. Murias, P., de Miguel, J. C., & Rodríguez, D. (2008). A composite indicator for university quality assessment: The case of Spanish higher education system. Social Indicators Research, 89(1), 129–146.

    Article  Google Scholar 

  49. Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Newbury Park: Sage Publications.

    Google Scholar 

  50. Muthén, B., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55(2), 463–469.

    Article  Google Scholar 

  51. Nagin, D. S., & Land, K. C. (1993). Age, criminal, careers, and population heterogeneity—Specification and estimation of a nonparametric, mixed Poisson model. Criminology, 31(3), 327–362.

    Article  Google Scholar 

  52. Nixon, E., Scullion, R., & Hearn, R. (2018). Her majesty the student: Marketised higher education and the narcissistic (dis)satisfactions of the student-consumer. Studies in Higher Education, 43(6), 927–943.

    Article  Google Scholar 

  53. Pennoni, F., & Romeo, I. (2017). Latent Markov and growth mixture models for ordinal individual responses with covariates: a comparison. Statistical Analysis and Data Mining: The ASA Data Science Journal, 10(1), 29–39.

    Article  Google Scholar 

  54. Ramaswamy, V., Desarbo, W. S., Reibstein, D. J., & Robinson, W. T. (1993). An empirical pooling approach for estimating marketing mix elasticities with PIMS data. Marketing Science, 12(1), 103–124.

    Article  Google Scholar 

  55. Rampichini, C., Grilli, L., & Petrucci, A. (2004). Analysis of university course evaluations: from descriptive measures to multilevel models. Statistical Methods and Applications, 13, 357–373.

    Article  Google Scholar 

  56. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.

    Article  Google Scholar 

  57. Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: the state of the art. Review of Educational Research, 83(4), 598–642.

    Article  Google Scholar 

  58. Stroebe, W. (2016). Why good teaching evaluations may reward bad teaching: on grade inflation and other unintended consequences of student evaluations. Perspectives on Psychological Science, 1(86), 800–816.

    Article  Google Scholar 

  59. Svinicki, M., & McKeachie, W. J. (2013). McKeachie’s teaching tips: Strategies, research, and theory for College and University teachers (13th ed.). Belmont: Wadsworth.

    Google Scholar 

  60. Theall, M., & Franklin, J. (Eds.). (2007). Student ratings of instruction: Issues for improving practice: New directions for teaching and learning. San Francisco: Jossey-Bass.

    Google Scholar 

  61. Uttl, B., White, C. A., & Gonzalez, D. W. (2016). Meta-analysis of faculty’s teaching effectiveness: student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42.

    Article  Google Scholar 

  62. van de Pol, F., & Langeheine, R. (1990). Mixed Markov latent class models. Sociological Methodology, 20, 213–247.

    Article  Google Scholar 

  63. van de Pol, F., & Mannan, H. (2002). Questions of a novice in latent Markov modelling. Methods of Psychological Research Online, 7(2), 1–18.

    Google Scholar 

  64. van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer.

    Google Scholar 

  65. Vermunt, J. K., Langeheine, R., & Bockenholt, U. (1999). Discrete-time discrete-state latent Markov models with time-constant and time-varying covariates. Journal of Educational and Behavioral Statistics, 24(2), 179–207.

    Article  Google Scholar 

  66. Wedel, M., & Kamakura, W. A. (2000). Market segmentation: Conceptual and methodological foundations. Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  67. Weng, L.-J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64(6), 956–972.

    Article  Google Scholar 

  68. Wiggins, L. M. (1955). Mathematical models for the interpretation of attitude and behavior change. Columbia: Columbia University.

    Google Scholar 

  69. Wu, H., & Leung, S.-O. (2017). Can Likert scales be treated as interval scales? A simulation study, Journal of Social Service Research, 43(4), 527–532.

    Google Scholar 

  70. Zabaleta, F. (2007). The use and misuse of student evaluations of teaching. Teaching in Higher Education, 12, 55–76.

    Article  Google Scholar 

  71. Zucchini, W., MacDonald, I. L., & Langrock, R. (2016). Hidden Markov models for time series: An introduction using R. Boca Raton: Chapman and Hall/CRC.

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the editor and three anonymous reviewers for their constructive comments, which helped us to improve the manuscript. This work was funded by the Portuguese Foundation for Science and Technology (Grant UID/GES/00315/2013 and UID/GES/00315/2019) and by Grant BIRD162088/16 financed by the University of Padua for the project entitled “Advances in Multilevel and Longitudinal Modelling”.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Francesca Bassi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Guerra, M., Bassi, F. & Dias, J.G. A Multiple-Indicator Latent Growth Mixture Model to Track Courses with Low-Quality Teaching. Soc Indic Res 147, 361–381 (2020). https://doi.org/10.1007/s11205-019-02169-x

Download citation

Keywords

  • Higher education
  • Quality of didactics
  • Latent growth mixture models
  • Outlier detection
  • Synthetic indicator
  • Data science