Skip to main content
SpringerLink
Log in

Introduction

  • Chapter
  • First Online:
A Chronicle of Permutation Statistical Methods

  • 1352 Accesses

Abstract

This chapter introduces permutation statistical methods. This first chapter begins with a brief description of the advantages of permutation methods from statisticians who are advocates of permutation tests, followed by a description of the methods of permutation tests including exact, moment-approximation, and resampling-approximation permutation tests. The chapter continues with an example that contrasts the well-known Student t test and results from exact, moment-approximation, and resampling-approximation permutation tests using historical data. The chapter concludes with a brief overview of the remaining chapters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The terms “permutation test” and “randomization test” are often used interchangeably.

  2. 2.

    In a reversal Tukey could not have predicted, at the time of this writing gold was trading at $1,775 per troy ounce, while platinum was only $1,712 per troy ounce [275].

  3. 3.

    Echoing Fisher’s argument that inference must be based solely on the data at hand [460], Haber refers to data dependency as “the data at hand principle” [565, p. 148].

  4. 4.

    Barton and David noted that it is desirable to make the minimum of assumptions, since, witness the oft-cited Bertrand paradox [163], that the assumptions made will often prejudice the conclusions reached [83, p. 455].

  5. 5.

    Rank-order statistics were among the earliest permutation tests, transforming the observed data into ranks, e.g., from smallest to largest. While they were an important step in the history of permutation tests, modern computing has superseded the need for rank-order tests in the majority of cases.

  6. 6.

    The complete data set is available in several formats at the Cambridge University Press site: http://uk.cambridge.org/resources/0521806631.

  7. 7.

    The diode and triode vacuum tubes were invented in 1906 and 1908, respectively, by Lee de Forest .

References

  1. Bakeman, R., Robinson, B.F., Quera, V.: Testing sequential association: Estimating exact p values using sampled permutations. Psychol. Methods 1, 4–15 (1996)

    Article  Google Scholar 

  2. Barnard, G.A.: A new test for 2 × 2 tables. Nature 156, 177 (1945)

    Article  MATH  MathSciNet  Google Scholar 

  3. Barton, D.E., David, F.N.: Randomization bases for multivariate tests I. The bivariate case: Randomness of n points in a plane. B. Int. Stat. Inst. 39, 455–467 (1961)

    Google Scholar 

  4. Bear, G.: Computationally intensive methods warrant reconsideration of pedagogy in statistics. Behav. Res. Methods Instrum. C 27, 144–147 (1995)

    Article  Google Scholar 

  5. Bergmann, R., Ludbrook, J., Spooren, W.P.J.M.: Different outcomes of the Wilcoxon–Mann–Whitney test from different statistics packages. Am. Stat. 54, 72–77 (2000)

    Google Scholar 

  6. Berry, K.J., Johnston, J.E., Mielke, P.W.: Permutation methods. Comput. Stat. 3, 527–542 (2011)

    Google Scholar 

  7. Berry, K.J., Mielke, P.W., Mielke, H.W.: The Fisher–Pitman permutation test: An attractive alternative to the F test. Psychol. Rep. 90, 495–502 (2002)

    Article  Google Scholar 

  8. Bertrand, J.L.F.: Calcul des Probabilitiés. Gauthier-Villars et fils, Paris (1889) [Reprinted by Chelsea Publishing (AMS), New York, in 1972]

    Google Scholar 

  9. Blaug, M.: The myth of the old Poor Law and the making of the new. J. Econ. Hist. 23, 151–184 (1963)

    Google Scholar 

  10. Box, J.F.: Gosset, Fisher, and the t distribution. Am. Stat. 35, 61–66 (1981)

    MathSciNet  Google Scholar 

  11. Boyer, G.R.: An Economic History of the English Poor Law: 1750–1850. Cambridge University Press, Cambridge (1990)

    Google Scholar 

  12. Bradley, J.V.: Distribution-free Statistical Tests. Prentice-Hall, Englewood Cliffs (1968)

    MATH  Google Scholar 

  13. Brillinger, D.R., Jones, L.V., Tukey, J.W.: The role of statistics in weather resources management. Tech. Rep. II, Weather Modification Advisory Board, United States Department of Commerce, Washington, DC (1978)

    Google Scholar 

  14. Constable, S.: When investing, try thinking outside the box. http://online.wsj.com/article/SB10001424052970203960804577241263821844868.html#mod=sunday_journal_primary_hs (26 February 2012). Accessed 29 Feb 2012

  15. Curran-Everett, D.: Explorations in statistics: Permutation methods. Adv. Physiol. Educ. 36, 181–187 (2012)

    Article  Google Scholar 

  16. Diaconis, P., Freedman, D.: Finite exchangeable sequences. Ann. Probab. 8, 745–764 (1980)

    Article  MATH  MathSciNet  Google Scholar 

  17. Donegani, M.: Asymptotic and approximate distribution of a statistic by resampling with or without replacement. Stat. Prob. Lett. 11, 181–183 (1991)

    Article  MATH  MathSciNet  Google Scholar 

  18. Dwass, M.: Modified randomization tests for nonparametric hypotheses. Ann. Math. Stat. 28, 181–187 (1957)

    Article  MATH  MathSciNet  Google Scholar 

  19. Eden, T., Yates, F.: On the validity of Fisher’s z test when applied to an actual example of non-normal data. J. Agric. Sci. 23, 6–17 (1933)

    Article  Google Scholar 

  20. Edgington, E.S.: Randomization Tests. Marcel Dekker, New York (1980)

    MATH  Google Scholar 

  21. Edgington, E.S.: Randomization Tests, 3rd edn. Marcel Dekker, New York (1995)

    MATH  Google Scholar 

  22. Edgington, E.S., Onghena, P.: Randomization Tests, 4th edn. Chapman & Hall/CRC, Boca Raton (2007)

    MATH  Google Scholar 

  23. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall/CRC, Boca Raton (1993)

    Book  MATH  Google Scholar 

  24. Eisenhart, C.: On the transition from “Student’s” z to “Student’s” t. Am. Stat. 33, 6–10 (1979)

    Google Scholar 

  25. Feinstein, A.R.: Clinical Biostatistics XXIII: The role of randomization in sampling, testing, allocation, and credulous idolatry (Part 2). Clin. Pharmacol. Ther. 14, 898–915 (1973)

    Google Scholar 

  26. Fisher, R.A.: Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh (1925)

    Google Scholar 

  27. Fisher, R.A.: The Design of Experiments. Oliver and Boyd, Edinburgh (1935)

    Google Scholar 

  28. Fisher, R.A.: ‘The coefficient of racial likeness’ and the future of craniometry. J. R. Anthropol. Inst. 66, 57–63 (1936)

    Google Scholar 

  29. Fisher, R.A.: Statistical Methods and Scientific Inference, 2nd edn. Hafner, New York (1959)

    Google Scholar 

  30. Frick, R.W.: Interpreting statistical testing: Process and propensity, not population and random sampling. Behav. Res. Methods Instrum. C 30, 527–535 (1998)

    Article  Google Scholar 

  31. Friedman, M.: A comparison of alternative tests of significance for the problem of m rankings. Ann. Math. Stat. 11, 86–92 (1940)

    Article  Google Scholar 

  32. Gabriel, K.R., Hall, W.J.: Rerandomization inference on regression and shift effects: Computationally feasible methods. J. Am. Stat. Assoc. 78, 827–836 (1983)

    Article  MATH  Google Scholar 

  33. Geary, R.C.: Some properties of correlation and regression in a limited universe. Metron 7, 83–119 (1927)

    Google Scholar 

  34. Good, I.J.: Further comments concerning the lady tasting tea or beer: P-values and restricted randomization. J. Stat. Comput. Simul. 40, 263–267 (1992)

    Article  Google Scholar 

  35. Good, P.I.: Permutation, Parametric and Bootstrap Tests of Hypotheses. Springer, New York (1994)

    Book  MATH  Google Scholar 

  36. Good, P.I.: Resampling Methods: A Practical Guide to Data Analysis. Birkhäuser, Boston (1999)

    Book  MATH  Google Scholar 

  37. Good, P.I.: Permutation, Parametric and Bootstrap Tests of Hypotheses, 2nd edn. Springer, New York (2000)

    Google Scholar 

  38. Good, P.I.: Resampling Methods: A Practical Guide to Data Analysis, 2nd edn. Birkhäuser, Boston (2001)

    Book  Google Scholar 

  39. Good, P.I.: Extensions of the concept of exchangeability and their applications. J. Mod. Appl. Stat. Methods 1, 243–247 (2002)

    Google Scholar 

  40. Haber, M.: Comments on “The test of homogeneity for 2 × 2 contingency tables: A review of and some personal opinions on the controversy” by G. Camilli. Psychol. Bull. 108, 146–149 (1990)

    Article  Google Scholar 

  41. Hall, P., Wilson, S.R.: Two guidelines for bootstrap hypothesis testing. Biometrics 47, 757–762 (1991)

    Article  MathSciNet  Google Scholar 

  42. Hayes, A.F.: Permutation test is not distribution-free: Testing H 0: ρ = 0. Psychol. Method. 1, 184–198 (1996)

    Article  Google Scholar 

  43. Holford, T.R.: Editorial: Exact methods for categorical data. Stat. Methods Med. Res. 12, 1 (2003)

    Article  MathSciNet  Google Scholar 

  44. Hooton, J.W.L.: Randomization tests: Statistics for experimenters. Comput. Methods Prog. Biomed. 35, 43–51 (1991)

    Article  Google Scholar 

  45. Hope, A.C.A.: A simplified Monte Carlo significance test procedure. J. R. Stat. Soc. B Met. 30, 582–598 (1968)

    MATH  Google Scholar 

  46. Hotelling, H., Pabst, M.R.: Rank correlation and tests of significance involving no assumption of normality. Ann. Math. Stat. 7, 29–43 (1936)

    Article  Google Scholar 

  47. Hubbard, R.: Alphabet soup: Blurring the distinctions between p’s and α’s in psychological research. Theor. Psychol. 14, 295–327 (2004)

    Article  Google Scholar 

  48. Hubert, L.: Assignment Methods in Combinatorial Data Analysis. Marcel Dekker, New York (1987)

    MATH  Google Scholar 

  49. Johnston, J.E., Berry, K.J., Mielke, P.W.: Permutation tests: Precision in estimating probability values. Percept. Motor Skill. 105, 915–920 (2007)

    Google Scholar 

  50. Johnston, J.E., Berry, K.J., Mielke, P.W.: Quantitative historical methods: A permutation alternative. Hist. Methods 42, 35–39 (2009)

    Article  Google Scholar 

  51. Kempthorne, O.: The randomization theory of experimental inference. J. Am. Stat. Assoc. 50, 946–967 (1955)

    MathSciNet  Google Scholar 

  52. Kempthorne, O.: Some aspects of experimental inference. J. Am. Stat. Assoc. 61, 11–34 (1966)

    Article  MathSciNet  Google Scholar 

  53. Kempthorne, O.: Why randomize? J. Stat. Plan. Infer. 1, 1–25 (1977)

    Article  MATH  MathSciNet  Google Scholar 

  54. Kennedy, P.E.: Randomization tests in econometrics. J. Bus. Econ. Stat. 13, 85–94 (1995)

    Google Scholar 

  55. Kingman, J.F.C.: Uses of exchangeability. Ann. Prob. 6, 183–197 (1978) [Abraham Wald memorial lecture delivered in August 1977 in Seattle, Washington]

    Google Scholar 

  56. Lachin, J.M.: Statistical properties of randomization in clinical trials. Control. Clin. Trials 9, 289–311 (1988)

    Article  Google Scholar 

  57. LaFleur, B.J., Greevy, R.A.: Introduction to permutation and resampling-based hypothesis tests. J. Clin. Child Adolesc. 38, 286–294 (2009)

    Article  Google Scholar 

  58. Ludbrook, J.: Advantages of permutation (randomization) tests in clinical and experimental pharmacology and physiology. Clin. Exp. Pharmacol. Physiol. 21, 673–686 (1994)

    Article  Google Scholar 

  59. Ludbrook, J.: Issues in biomedical statistics: Comparing means by computer-intensive tests. Aust. N.Z. J. Surg. 65, 812–819 (1995)

    Google Scholar 

  60. Ludbrook, J., Dudley, H.A.F.: Issues in biomedical statistics: Analyzing 2 × 2 tables of frequencies. Aust. N. Z. J. Surg. 64, 780–787 (1994)

    Article  Google Scholar 

  61. Ludbrook, J., Dudley, H.A.F.: Why permutation tests are superior to t and F tests in biomedical research. Am. Stat. 52, 127–132 (1998)

    Google Scholar 

  62. Lunneborg, C.E.: Data Analysis by Resampling: Concepts and Applications. Duxbury, Pacific Grove (2000)

    Google Scholar 

  63. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology. Chapman & Hall, London (1991)

    Book  MATH  Google Scholar 

  64. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology, 2nd edn. Chapman & Hall, London (1997)

    MATH  Google Scholar 

  65. Manly, B.F.J.: Randomization, Bootstrap and Monte Carlo Methods in Biology, 3rd edn. Chapman & Hall/CRC, Boca Raton (2007)

    MATH  Google Scholar 

  66. Mehta, C.R., Patel, N.R.: A network algorithm for the exact treatment of the 2 × k contingency table. Commun. Stat. Simul. C 9, 649–664 (1980)

    Article  MathSciNet  Google Scholar 

  67. Mielke, P.W., Berry, K.J.: Permutation Methods: A Distance Function Approach. Springer, New York (2001)

    Book  Google Scholar 

  68. Mielke, P.W., Berry, K.J.: Permutation Methods: A Distance Function Approach, 2nd edn. Springer, New York (2007)

    Google Scholar 

  69. Mielke, P.W., Berry, K.J., Johnston, J.E.: Unweighted and weighted kappa as measures of agreement for multiple judges. Int. J. Manag. 26, 213–223 (2009)

    Google Scholar 

  70. Mielke, P.W., Berry, K.J., Johnston, J.E.: Robustness without rank order statistics. J. Appl. Stat. 38, 207–214 (2011)

    Article  MathSciNet  Google Scholar 

  71. Mielke, P.W., Iyer, H.K.: Permutation techniques for analyzing multi-response data from randomized block experiments. Commun. Stat. Theor. Methods 11, 1427–1437 (1982)

    Article  MATH  Google Scholar 

  72. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika 20A, 175–240 (1928)

    Google Scholar 

  73. Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference: Part II. Biometrika 20A, 263–294 (1928)

    Google Scholar 

  74. Noreen, E.W.: Computer-intensive Methods For Testing Hypotheses: An Introduction. Wiley, New York (1989)

    Google Scholar 

  75. Pesarin, F., Salmaso, L.: Permutation Tests for Complex Data: Theory, Applications and Software. Wiley, Chichester (2010)

    Book  Google Scholar 

  76. Pitman, E.J.G.: Significance tests which may be applied to samples from any populations. Suppl. J. R. Stat. Soc. 4, 119–130 (1937)

    Article  Google Scholar 

  77. Pitman, E.J.G.: Significance tests which may be applied to samples from any populations: III. The analysis of variance test. Biometrika 29, 322–335 (1938)

    MATH  Google Scholar 

  78. Read, T.R.C., Cressie, N.A.C.: Goodness-of-Fit for Discrete Multivariate Data. Springer, New York (1988)

    Book  MATH  Google Scholar 

  79. Romano, J.P.: Bootstrap and randomization tests of some nonparametric hypotheses. Ann. Stat. 17, 141–159 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  80. Sakaori, F.: Permutation test for equality of correlation coefficients in two populations. Commun. Stat. Simul. C 31, 641–651 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  81. Scheffé, H.: The Analysis of Variance. Wiley, New York (1959)

    MATH  Google Scholar 

  82. Simon, J.L.: Resampling: The New Statistics. Duxbury, Pacific Grove (1997)

    Google Scholar 

  83. Spława-Neyman, J.: Próba uzasadnienia zastosowań rachunku prawdopodobieństwa do doświadczeń polowych (On the application of probability theory to agricultural experiments. Essay on principles. Section 9). Rocz. Nauk Rolnicz. (Ann. Agric. Sci.) 10, 1–51 (1923) [Translated from the original Polish by D. M. Dabrowska and T. P. Speed and published in Stat. Sci. 5, 465–472 (1990)]

    Google Scholar 

  84. “Student”: The probable error of a mean. Biometrika 6, 1–25 (1908) [“Student” is a nom de plume for William Sealy Gosset]

    Google Scholar 

  85. Tukey, J.W.: Bias and confidence in not-quite large samples. Ann. Math. Stat. 29, 614 (1958)

    Article  Google Scholar 

  86. Tukey, J.W.: Tightening the clinical trial. Control. Clin. Trials 14, 266–285 (1993)

    Article  Google Scholar 

  87. Tukey, J.W.: Randomization and re-randomization: The wave of the past in the future. In: Statistics in the Pharmaceutical Industry: Past, Present and Future. Philadelphia Chapter of the American Statistical Association (June 1988) [Presented at a Symposium in Honor of Joseph L. Ciminera held in June 1988 at Philadelphia, Pennsylvania]

    Google Scholar 

  88. Westfall, P.H., Young, S.S.: Resampling-based Multiple Testing: Examples and Methods for p-value Adjustment. Wiley, New York (1993)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Sociology, Colorado State University, Fort Collins, CO, USA

    Kenneth J. Berry

  2. U.S. Government, Alexandria, VA, USA

    Janis E. Johnston

  3. Department of Statistics, Colorado State University, Fort Collins, CO, USA

    Paul W. Mielke Jr.

Authors
  1. Kenneth J. Berry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janis E. Johnston
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul W. Mielke Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Berry, K.J., Johnston, J.E., Mielke, P.W. (2014). Introduction. In: A Chronicle of Permutation Statistical Methods. Springer, Cham. https://doi.org/10.1007/978-3-319-02744-9_1

Download citation

Publish with us

Policies and ethics

Search

Navigation