Nonparametric MANOVA in meaningful effects

  • Dennis DoblerEmail author
  • Sarah Friedrich
  • Markus Pauly


Multivariate analysis of variance (MANOVA) is a powerful and versatile method to infer and quantify main and interaction effects in metric multivariate multi-factor data. It is, however, neither robust against change in units nor meaningful for ordinal data. Thus, we propose a novel nonparametric MANOVA. Contrary to existing rank-based procedures, we infer hypotheses formulated in terms of meaningful Mann–Whitney-type effects in lieu of distribution functions. The tests are based on a quadratic form in multivariate rank effect estimators, and critical values are obtained by bootstrap techniques. The newly developed procedures provide asymptotically exact and consistent inference for general models such as the nonparametric Behrens–Fisher problem and multivariate one-, two-, and higher-way crossed layouts. Computer simulations in small samples confirm the reliability of the developed method for ordinal and metric data with covariance heterogeneity. Finally, an analysis of a real data example illustrates the applicability and correct interpretation of the results.


Covariance heteroscedasticity Multivariate data Multivariate ordinal data Multiple samples Rank-based methods Wild bootstrap 



This work was supported by the German Research Foundation (Grant No. PA-2409 4-1).

Supplementary material

10463_2019_717_MOESM1_ESM.pdf (211 kb)
Supplementary material 1 (pdf 211 KB)


  1. Acion, L., Peterson, J. J., Temple, S., Arndt, S. (2006). Probabilistic index: An intuitive non-parametric approach to measuring the size of treatment effects. Statistics in Medicine, 25(4), 591–602.MathSciNetCrossRefGoogle Scholar
  2. Akritas, M. G. (1986). Bootstrapping the Kaplan–Meier estimator. Journal of the American Statistical Association, 81(396), 1032–1038.MathSciNetzbMATHGoogle Scholar
  3. Akritas, M. G., Arnold, S. F., Brunner, E. (1997). Nonparametric hypotheses and rank statistics for unbalanced factorial designs. Journal of the American Statistical Association, 92(437), 258–265.MathSciNetCrossRefzbMATHGoogle Scholar
  4. Barbiero, A., Ferrari, P. A. (2015). GenOrd: Simulation of discrete random variables with given correlation matrix and marginal distributions. R package version 1.4.0.Google Scholar
  5. Bathke, A. C., Harrar, S. W., Madden, L. V. (2008). How to compare small multivariate samples using nonparametric tests. Computational Statistics and Data Analysis, 52(11), 4951–4965.MathSciNetCrossRefzbMATHGoogle Scholar
  6. Bathke, A. C., Friedrich, S., Pauly, M., Konietschke, F., Staffen, W., Strobl, N., Höller, Y. (2018). Testing mean differences among groups: Multivariate and repeated measures analysis with minimal assumptions. Multivariate Behavioral Research, 53(3), 348–359.CrossRefGoogle Scholar
  7. Brown, B. M., Hettmansperger, T. P. (2002). Kruskal–Wallis, multiple comparisons and Efron dice. Australian and New Zealand Journal of Statistics, 44(4), 427–438.MathSciNetCrossRefzbMATHGoogle Scholar
  8. Brumback, L. C., Pepe, M. S., Alonzo, T. A. (2006). Using the ROC curve for gauging treatment effect in clinical trials. Statistics in Medicine, 25(4), 575–590.MathSciNetCrossRefGoogle Scholar
  9. Brunner, E., Puri, M. L. (2001). Nonparametric methods in factorial designs. Statistical Papers, 42(1), 1–52.MathSciNetCrossRefzbMATHGoogle Scholar
  10. Brunner, E., Dette, H., Munk, A. (1997). Box-type approximations in nonparametric factorial designs. Journal of the American Statistical Association, 92(440), 1494–1502.MathSciNetCrossRefzbMATHGoogle Scholar
  11. Brunner, E., Munzel, U., Puri, M. L. (2002). The multivariate nonparametric Behrens–Fisher problem. Journal of Statistical Planning and Inference, 108(1), 37–53.MathSciNetCrossRefzbMATHGoogle Scholar
  12. Brunner, E., Konietschke, F., Pauly, M., Puri, M. L. (2017). Rank-based procedures in factorial designs: Hypotheses about non-parametric treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(5), 1463–1485.Google Scholar
  13. Brunner, E., Konietschke, F., Bathke, A. C., Pauly, M. (2018). Ranks and pseudo-ranks: Paradoxical results of rank tests. arXiv preprint arXiv:1802.05650.Google Scholar
  14. Davidson, R., Flachaire, E. (2008). The wild bootstrap, tamed at last. Journal of Econometrics, 146(1), 162–169.MathSciNetCrossRefzbMATHGoogle Scholar
  15. De Neve, J., Thas, O. (2015). A regression framework for rank tests based on the probabilistic index model. Journal of the American Statistical Association, 110(511), 1276–1283.MathSciNetCrossRefzbMATHGoogle Scholar
  16. Dobler, D. (2017). A discontinuity adjustment for subdistribution function confidence bands applied to right-censored competing risks data. Electronic Journal of Statistics, 11(2), 3673–3702.MathSciNetCrossRefzbMATHGoogle Scholar
  17. Dobler, D., Pauly, M. (2018). Inference for the Mann–Whitney effect for right-censored and tied data. TEST, 27(3), 639–658.MathSciNetCrossRefGoogle Scholar
  18. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26.MathSciNetCrossRefzbMATHGoogle Scholar
  19. Efron, B. (1981). Censored data and the bootstrap. Journal of the American Statistical Association, 76(374), 312–319.MathSciNetCrossRefzbMATHGoogle Scholar
  20. Ellis, A. R., Burchett, W. W., Harrar, S. W., Bathke, A. C. (2017). Nonparametric inference for multivariate data: The R package npmv. Journal of Statistical Software, 76(4), 1–18.Google Scholar
  21. Ferrari, P. A., Barbiero, A. (2012). Simulating ordinal data. Multivariate Behavioral Research, 47(4), 566–589.CrossRefGoogle Scholar
  22. Friedrich, S., Pauly, M. (2018). Mats: Inference for potentially singular and heteroscedastic manova. Journal of Multivariate Analysis, 165, 166–179.MathSciNetCrossRefzbMATHGoogle Scholar
  23. Friedrich, S., Konietschke, F., Pauly, M. (2017). A wild bootstrap approach for nonparametric repeated measurements. Computational Statistics and Data Analysis, 113, 38–52.MathSciNetCrossRefzbMATHGoogle Scholar
  24. Gao, X., Alvo, M. (2005). A unified nonparametric approach for unbalanced factorial designs. Journal of the American Statistical Association, 100(471), 926–941.MathSciNetCrossRefzbMATHGoogle Scholar
  25. Gao, X., Alvo, M. (2008). Nonparametric multiple comparison procedures for unbalanced two-way layouts. Journal of Statistical Planning and Inference, 138(12), 3674–3686.MathSciNetCrossRefzbMATHGoogle Scholar
  26. Gao, X., Alvo, M., Chen, J., Li, G. (2008). Nonparametric multiple comparison procedures for unbalanced one-way factorial designs. Journal of Statistical Planning and Inference, 138(8), 2574–2591.MathSciNetCrossRefzbMATHGoogle Scholar
  27. Halvorsen, K. B. (2015). ElemStatLearn: Data sets, functions and examples from the book: The elements of statistical learning, data mining, inference, and prediction” by Trevor Hastie, Robert Tibshirani and Jerome Friedman. R package version 2015.6.26.Google Scholar
  28. Harrar, S. W., Bathke, A. C. (2008). Nonparametric methods for unbalanced multivariate data and many factor levels. Journal of Multivariate Analysis, 99(8), 1635–1664.MathSciNetCrossRefzbMATHGoogle Scholar
  29. Harrar, S. W., Bathke, A. C. (2012). A modified two-factor multivariate analysis of variance: Asymptotics and small sample approximations (and erratum). Annals of the Institute of Statistical Mathematics, 64(1), 135–165.MathSciNetCrossRefzbMATHGoogle Scholar
  30. Kieser, M., Friede, T., Gondan, M. (2013). Assessment of statistical significance and clinical relevance. Statistics in Medicine, 32(10), 1707–1719.MathSciNetCrossRefGoogle Scholar
  31. Konietschke, F., Hothorn, L. A., Brunner, E. (2012). Rank-based multiple test procedures and simultaneous confidence intervals. Electronic Journal of Statistics, 6, 738–759.MathSciNetCrossRefzbMATHGoogle Scholar
  32. Konietschke, F., Bathke, A. C., Harrar, S. W., Pauly, M. (2015). Parametric and nonparametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291–301.MathSciNetCrossRefzbMATHGoogle Scholar
  33. Kruskal, W. H. (1952). A nonparametric test for the several sample problem. The Annals of Mathematical Statistics, 23(4), 525–540.MathSciNetCrossRefzbMATHGoogle Scholar
  34. Kruskal, W. H., Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621.CrossRefzbMATHGoogle Scholar
  35. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.CrossRefGoogle Scholar
  36. Mann, H. B., Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60.MathSciNetCrossRefzbMATHGoogle Scholar
  37. Marcus, R., Eric, P., Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3), 655–660.MathSciNetCrossRefzbMATHGoogle Scholar
  38. Munzel, U. (1999). Linear rank score statistics when ties are present. Statistics and Probability Letters, 41(4), 389–395.MathSciNetCrossRefzbMATHGoogle Scholar
  39. Munzel, U., Brunner, E. (2000). Nonparametric methods in multivariate factorial designs. Journal of Statistical Planning and Inference, 88(1), 117–132.MathSciNetCrossRefzbMATHGoogle Scholar
  40. Puri, M. L., Sen, P. K. (1971). Nonparametric methods in multivariate analysis. New York: Wiley.zbMATHGoogle Scholar
  41. R Core Team. (2016). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.Google Scholar
  42. Rust, S. W., Filgner, M. A. (1984). A modification of the Kruskal–Wallis statistic for the generalized Behrens–Fisher problem. Communications in Statistics-Theory and Methods, 13(16), 2013–2027.MathSciNetCrossRefzbMATHGoogle Scholar
  43. Ruymgaart, F. H. (1980). A unified approach to the asymptotic distribution theory of certain midrank statistics. In J.-P. Raoult (Ed.), Statistique non Parametrique Asymptotique, pp. 1–18. Berlin: Springer.Google Scholar
  44. Thangavelu, K., Brunner, E. (2007). Wilcoxon–Mann–Whitney test for stratified samples and Efron’s paradox dice. Journal of Statistical Planning and Inference, 137(3), 720–737. Special Issue on Nonparametric Statistics and Related Topics: In honor of M.L. Puri.Google Scholar
  45. Thas, O., De Neve, J., Clement, L., Ottoy, J.-P. (2012). Probabilistic index models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(4), 623–671.Google Scholar
  46. Umlauft, M., Konietschke, F., Pauly, M. (2017). Rank-based permutation approaches for nonparametric factorial designs. British Journal of Mathematical and Statistical Psychology, 70, 368–390.CrossRefzbMATHGoogle Scholar
  47. Umlauft, M., Placzek, M., Konietschke, F., Pauly, M. (2019). Wild bootstrapping rank-based procedures: Multiple testing in nonparametric factorial repeated measures designs. Journal of Multivariate Analysis, 171, 176–192.MathSciNetCrossRefGoogle Scholar
  48. van der Vaart, A. W., Wellner, J. A. (1996). Weak convergence and empirical processes. New York: Springer.CrossRefzbMATHGoogle Scholar
  49. Wu, C.-F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. The Annals of Statistics, 14(4), 1261–1295.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© The Institute of Statistical Mathematics, Tokyo 2019

Authors and Affiliations

  1. 1.Department of MathematicsVrije Universiteit AmsterdamAmsterdamThe Netherlands
  2. 2.Section of BiostatisticsUniversity of CopenhagenCopenhagenDenmark
  3. 3.Institute for Mathematical Statistics and Industrial Applications, Faculty of StatisticsTechnical University of DortmundDortmundGermany

Personalised recommendations