Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Beyond 2000

  • Chapter
  • First Online:
A Chronicle of Permutation Statistical Methods

Abstract

This chapter chronicles the period from 2001 to 2010. By the beginning of this period, permutation statistical methods had come of age and advances were comprised more of application and expansion into new fields and disciplines than the development of new permutation methods that characterized earlier years. In this period computing power was sufficient to accommodate the needs of computational statisticians utilizing permutation statistical methods, including both exact and Monte Carlo permutation tests.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    One petaflops indicates a quadrillion operations per second, or a 1 with 15 zeroes following it.

  2. 2.

    One exoflops indicates a quintillion floating operations per second, or a 1 with 18 zeroes after it.

  3. 3.

    The Fisher Iris data is a multivariate data set analyzed by Fisher in 1936 to illustrate discriminate analysis [454]. The data were collected and originally published by Edgar Anderson in 1935 and 1936 [17, 18].

  4. 4.

    Although there are no references in the letter by Good , it is readily apparent that Good was basing his criticisms on an article by Weinberg and Lagakos titled “Efficiency comparisons of rank and permutation tests based on summary statistics computed from repeated-measures data,” which was published in Statistics in Medicine in 2001 [1425].

  5. 5.

    The first letter was published in Statistics in Medicine in 2004 [529].

  6. 6.

    It should be noted that Good took a position on the use of rank tests in stark contrast to other researchers of the time, many of whom were abandoning rank tests in favor of permutation tests using the original raw score measurements instead of converting raw scores to rank-order statistics (q.v. page 402).

  7. 7.

    The small ruminants studied by Önder were purebred Ile de France sheep and crossbred Chios and Awassi sheep.

  8. 8.

    Technically, Bray–Curtis is a dissimilarity measure, not a distance measure, as it does not satisfy the triangle inequality (q.v. page 255).

  9. 9.

    Agresti’s “exact conditional approach” corresponds to marginal frequency totals that are fixed, while his “exact unconditional approach” corresponds to marginal frequency totals that are not fixed.

  10. 10.

    See also an informative 2011 article by Yung-Pin Chen comparing the chi-squared and Fisher’s exact probability tests in The American Statistician [251].

  11. 11.

    See also a comprehensive review of the use of the mid-P procedure by Berry and Armitage in The Statistician, published in 1995 [107].

  12. 12.

    In this section, Cohen’s kappa is indicated by κ c to distinguish it from the κ n of Brennan and Prediger .

  13. 13.

    The oribatid mite is considered to be the world’s strongest animal, able to support 1,180 times its weight. By contrast, the strongest human can support approximately three times its weight.

  14. 14.

    Here, the number of species (p) is the number of judges or raters in the usual implementation of Kendall’s coefficient of concordance.

  15. 15.

    For more on the Hellinger and other transformations of species data, see a 2001 article by Legendre and Gallagher [811].

  16. 16.

    Note that, in contrast to an exact permutation test, a resampling-approximation permutation test does not require calculation of a hypergeometric probability value for each possible arrangement of cell frequencies.

  17. 17.

    Linear weighting was first suggested by Cicchetti and Allison [255]. In 2008 Vanbelle and Albert demonstrated that weighted kappa for m = 2 independent raters and r ≥ 3 ordered categories is equivalent to deriving the weighted kappa coefficient from unweighted kappa values computed on r − 1 embedded 2 × 2 classification tables, given linear weighting [1393]. In 2009 Mielke and Berry generalized the results of Vanbelle and Albert to m ≥ 2 independent raters [967].

  18. 18.

    In 1972 Cohen admitted that the formulae for the approximate variance of weighted kappa given by Cohen in 1968 [264] and by Everitt in 1968 [415] were both incorrect [265, p. 64], but that the formula given by Fleiss , Cohen , and Everitt in 1969 [469] was, in fact, correct. This latter statement turned out to be incorrect.

  19. 19.

    In 2009 Mielke , Berry , and Johnston provided an example analysis based on m = 4 independent raters for both unweighted and weighted kappa that was published in International Journal of Management [977].

  20. 20.

    The three research designs were first described by Barnard in an article in Biometrika in 1947 [67] (q.v. page 130).

  21. 21.

    For an excellent discussion of the chi-squared test for 2 × 2 contingency tables, see a 1990 article by John Richardson in British Journal of Mathematical and Statistical Psychology [1170].

  22. 22.

    Karl Pearson had miscalculated the degrees of freedom in 1900 and it was corrected by Fisher in 1922, which did little to improve their antagonistic relationship.

  23. 23.

    As Egon Pearson noted in 1947, the correction for continuity utilized by Yates in 1934 was not new at the time, having been used by statisticians for many years prior when employing a normal or skew curve to give the sum of terms of a binomial or hypergeometric series [1095, p. 147].

  24. 24.

    On this topic, see articles by Barnard [65], Mielke and Berry [947, 948], Richardson [1170, 1171], Berry and Mielke [136], Upton [1385], and Schouten , Molenaar , van Strik , and Boomsma [1237].

  25. 25.

    Campbell , unfortunately, neglected to note that the test was also independently developed by Yates in 1934 (q.v. page 43).

  26. 26.

    For reasons why the assumption of normality is critical in conventional statistical analyses, see a 2011 paper by Mordkoff [1006].

  27. 27.

    Emphasis in the original.

  28. 28.

    See also a short but comprehensive 2010 article on this topic by Tom Siegfried in Science News [1274].

  29. 29.

    In 1907, the Italian economist and sociologist Vilfredo Pareto created a mathematical formula to describe the unequal distribution of wealth in Italy, observing that 20 % of the people owned 80 % of the wealth [1087]. This became known as the Pareto Principle or Pareto’s Law. In general, the 80/20 rule has come to mean that in anything, a few (20 %) are vital and many (80 %) are trivial.

  30. 30.

    There is a counter argument, of course, that power may not be lost when converting raw scores to ranks, and may even be increased, depending on which assumptions are violated and in which manner; see for example articles by Blair and Higgins [168], Higgins and Blair [614], Good [530], Hodges and Lehmann [635], Keller-McNulty and Higgins , [714], Lehmann [815], and van den Brink and van den Brink [1389].

  31. 31.

    The exact Fisher–Pitman two-sample permutation tests with v = 1, 2 and the exact permutation version of the Wilcoxon–Mann–Whitney two-sample rank-sum test are specific forms of MRPP (q.v. page 249) with g = 2 and r = 1 (q.v. page 256).

  32. 32.

    The exact Fisher–Pitman matched-pairs permutation tests with v = 1, 2 and the exact permutation version of the Wilcoxon matched-pairs rank-sum test are specific forms of MRBP with g = 2 and r = 1 (qq.v. pages 310 and 317).

  33. 33.

    In 2008 Malcolm Gladwell published an entire book titled Outliers, in which he defines an outlier as “a statistical observation that is markedly different in value from the others of the sample” [515, p. 3].

  34. 34.

    Two lucid discussions of outliers and how to treat them are contained in papers by David Finney in 2006 [438] and John Ludbrook in 2008 [853].

  35. 35.

    Kruskal suggested that only identified outliers be given a lesser weight, then proceeding as usual [777]. Others who earlier suggested the weighting of outliers include S. Newcomb [1032] who suggested that each observation be weighted by its residual, E.G. Stone [1325], and F.Y. Edgeworth [380, 393].

  36. 36.

    Others have analyzed the Westlund and Kurland data, including Jolayemi in 1990 [698] and Borkowf in 2004 [183].

  37. 37.

    Note that Brusco et al., in order to ensure an efficient procedure, began the process by filling in those cells with the smallest weights; in this case, following Eq. (6.4), \(w_{\mathit{ij}} = w_{1,4} = 1 - {(1 - 4)}^{2}/{(4 - 1)}^{2} = w_{4,1} = 1 - {(4 - 1)}^{2}/{(4 - 1)}^{2} = 0\).

  38. 38.

    Actually, 560 = 867, 361, 737, 988, 403, 547, 205, 962, 240, 695, 953, 369, 140, 625.

  39. 39.

    This was the eighth article by Curran-Everett in a series published under the rubric “Explorations in statistics” in Advances in Physiology Education; the previous articles in the series covered standard deviations and standard errors, confidence intervals, hypothesis tests, the bootstrap, correlation, power, and regression, and were published in 2008, 2009, 2010, and 2011 [300–306].

References

  1. Agresti, A.: Exact inference for categorical data: Recent advances and continuing controversies. Stat. Med. 20, 2709–2722 (2001)

    Google Scholar 

  2. Algina, J., Keselman, H.J., Penfield, R.D.: Confidence intervals for an effect size measure in multiple linear regression. Educ. Psychol. Meas. 67, 207–218 (2007)

    MathSciNet  Google Scholar 

  3. Algina, J., Keselman, H.J., Penfield, R.D.: Confidence intervals for squared semipartial correlation coefficients: The effect of nonnormality. Educ. Psychol. Meas. 70, 926–940 (2010)

    Google Scholar 

  4. Anderson, E.: The Irises of the Gaspé peninsular. Bull. Am. Iris Soc. 59, 2–5 (1935)

    Google Scholar 

  5. Anderson, E.: The species problem in Iris. Ann. Mo. Bot. Gdn. 23, 457–509 (1936)

    Google Scholar 

  6. Anderson, M.J., ter Braak, C.J.F.: Permutation tests for multi-factorial analysis of variance. J. Stat. Comput. Simul. 73, 85–113 (2003)

    MATH  MathSciNet  Google Scholar 

  7. Anderson, M.J., Legendre, P.: An empirical comparison of permutation methods for tests of partial regression coefficients in a linear model. J. Stat. Comput. Simul. 62, 271–303 (1999)

    MATH  MathSciNet  Google Scholar 

  8. Anderson, M.J., Robinson, J.: Permutation tests for linear models. Aust. N. Z. J. Stat. 43, 75–88 (2001)

    MATH  MathSciNet  Google Scholar 

  9. Andriani, P., McKelvey, B.: Perspective — from Gaussian to Paretian thinking: Causes and implications of power laws in organizations. Organ. Sci. 20, 1053–1071 (2009)

    Google Scholar 

  10. Ansari, A.R., Bradley, R.A.: Rank sum tests for dispersion. Ann. Math. Stat. 31, 1174–1189 (1960)

    MATH  MathSciNet  Google Scholar 

  11. Anscombe, F.J.: Rejection of outliers. Technometrics 2, 123–147 (1960)

    MATH  MathSciNet  Google Scholar 

  12. Arboretti Giancristofaro, R., Bonnini, S., Pesarin, F.: A permutation approach for testing heterogeneity in two-sample categorical variables. Stat. Comput. 19, 209–216 (2009)

    MathSciNet  Google Scholar 

  13. Arbuckle, J., Aiken, L.S.: A program for Pitman’s permutation test for differences in location. Behav. Res. Methods Instrum. 7, 381 (1975)

    Google Scholar 

  14. Bailer, A.J.: Testing variance equality with randomization tests. J. Stat. Comput. Simul. 31, 1–8 (1989)

    MATH  Google Scholar 

  15. Banerjee, M., Capozzoli, M., McSweeney, L., Sinha, D.: Beyond kappa: A review of interrater agreement measures. Can. J. Stat. 27, 3–23 (1999)

    MATH  MathSciNet  Google Scholar 

  16. Barboza, D., Markoff, J.: Power in numbers: China aims for high-tech primacy. NY Times 161, D2–D3 (6 December 2011)

    Google Scholar 

  17. Barnard, G.A.: A new test for 2 × 2 tables. Nature 156, 177 (1945)

    MATH  MathSciNet  Google Scholar 

  18. Barnard, G.A.: 2 × 2 tables. A note on E. S. Pearson’s paper. Biometrika 34, 168–169 (1947)

    MathSciNet  Google Scholar 

  19. Barnard, G.A.: Significance tests for 2 × 2 tables. Biometrika 34, 123–138 (1947)

    MATH  MathSciNet  Google Scholar 

  20. Bennett, E.M., Alpert, R., Goldstein, A.C.: Communications through limited-response questioning. Public Opin. Quart. 18, 303–308 (1954)

    Google Scholar 

  21. Bernardin, H.J., Beatty, R.W.: Performance Appraisal: Assessing Human Behavior at Work. Kent, Boston (1984)

    Google Scholar 

  22. Bernoulli, D.: Indicatio maxime probabilis plurium observationum discrepantium atque verisimilluma inductio inde formanda (The most probable choice between several discrepant observations and the formation therefrom of the most likely induction). Acta Acad. Sci. Petropol. 1, 1–33 (1777) [See the English translation by C.G. Allen in Biometrika 48, 1–18 (1961)]

    Google Scholar 

  23. Berry, G., Armitage, P.: Mid-P confidence intervals: A brief review. Statistician 44, 417–423 (1995)

    Google Scholar 

  24. Berry, K.J., Johnston, J.E., Mielke, P.W.: Exact and resampling probability values for weighted kappa. Psychol. Rep. 96, 243–252 (2005)

    Google Scholar 

  25. Berry, K.J., Johnston, J.E., Mielke, P.W.: Exact and resampling probability values for measures associated with ordered R by C contingency tables. Psychol. Rep. 99, 231–238 (2006)

    Google Scholar 

  26. Berry, K.J., Johnston, J.E., Mielke, P.W.: Exact permutation probability values for weighted kappa. Psychol. Rep. 102, 53–57 (2008)

    Google Scholar 

  27. Berry, K.J., Johnston, J.E., Mielke, P.W.: Weighted kappa for multiple raters. Percept. Motor Skill. 107, 837–848 (2008)

    Google Scholar 

  28. Berry, K.J., Johnston, J.E., Mielke, P.W.: Analysis of trend: A permutation alternative to the F test. Percept. Motor Skill. 112, 247–257 (2011)

    Google Scholar 

  29. Berry, K.J., Johnston, J.E., Mielke, P.W.: Permutation methods. Comput. Stat. 3, 527–542 (2011)

    Google Scholar 

  30. Berry, K.J., Mielke, P.W.: Analyzing independence in r-way contingency tables. Educ. Psychol. Meas. 49, 605–607 (1989)

    Google Scholar 

  31. Berry, K.J., Mielke, P.W.: Nonasymptotic significance tests for two measures of agreement. Percept. Motor Skill. 93, 109–114 (2001)

    Google Scholar 

  32. Berry, K.J., Mielke, P.W., Mielke, H.W.: The Fisher–Pitman permutation test: An attractive alternative to the F test. Psychol. Rep. 90, 495–502 (2002)

    Google Scholar 

  33. Blair, R.C., Higgins, J.J.: A comparison of the power of Wilcoxon’s rank-sum statistic to that of Student’s t under various nonnormal distributions. J. Educ. Stat. 5, 309–335 (1980)

    Google Scholar 

  34. Boik, R.J.: The Fisher–Pitman permutation test: A non-robust alternative to the normal theory F test when variances are heterogeneous. Br. J. Math. Stat. Psychol. 40, 26–42 (1987)

    MATH  MathSciNet  Google Scholar 

  35. Borgatta, E.F.: My student, the purist: A lament. Sociol. Quart. 9, 29–34 (1968)

    Google Scholar 

  36. Borkowf, C.B.: An efficient algorithm for generating two-way contingency tables with fixed marginal totals and arbitrary mean proportions, with applications to permutation tests. Comput. Stat. Data Anal. 44, 431–449 (2004)

    MathSciNet  Google Scholar 

  37. Boyett, J.M.: Algorithm 144: R × C tables with given row and column totals. J. R. Stat. Soc. C Appl. Stat. 28, 329–332 (1979)

    MATH  Google Scholar 

  38. Bradley, J.V.: A common situation conducive to bizarre distribution shapes. Am. Stat. 31, 147–150 (1977)

    Google Scholar 

  39. Bray, J.R., Curtis, J.T.: An ordination of the upland forest communities of southern Wisconsin. Ecol. Monogr. 27, 326–349 (1957)

    Google Scholar 

  40. Brennan, R.L., Prediger, D.J.: Coefficient kappa: Some uses, misuses, and alternatives. Educ. Psychol. Meas. 41, 687–699 (1981)

    Google Scholar 

  41. Bross, I.D.J.: Is there an increased risk? Fed. Proc. 13, 815–819 (1954)

    Google Scholar 

  42. Bross, I.D.J.: How to use ridit analysis. Biometrics 14, 18–38 (1958)

    Google Scholar 

  43. Brown, B.M., Maritz, J.S.: Distribution-free methods in regression. Aust. J. Stat. 24, 318–331 (1982)

    MATH  MathSciNet  Google Scholar 

  44. Brusco, M.J., Stahl, S., Steinley, D.: An implicit enumeration method for an exact test of weighted kappa. Br. J. Math. Stat. Psychol. 61, 439–452 (2008)

    MathSciNet  Google Scholar 

  45. Cade, B.S., Richards, J.D.: A permutation test for quantile regression. J. Agric. Biol. Environ. Sci. 11, 106–126 (2006)

    Google Scholar 

  46. Campbell, I.: Chi-squared and Fisher–Irwin tests of two-by-two tables with small sample recommendations. Stat. Med. 26, 3661–3675 (2007)

    MathSciNet  Google Scholar 

  47. Case, L.: Intel’s Ivy Bridge processor: Leaner and meaner. http://www.pcadvisor.co.uk/news/pc-components/3353194/intels-ivy-bridge-processor-leaner-meaner/ (23 April 2012). Accessed 29 Apr 2012

  48. Chen, Y.P.: Do the chi-square test and Fisher’s exact test agree in determining extreme for 2 × 2 tables? Am. Stat. 65, 239–245 (2011)

    Google Scholar 

  49. Chihara, L.M., Hesterberg, T.C.: Mathematical Statistics with Resampling and R. Wiley, New York (2011)

    MATH  Google Scholar 

  50. Cicchetti, D., Allison, A.: A new procedure for assessing reliability of scoring EEG sleep recordings. Am. J. EEG Technol. 11, 101–109 (1971)

    Google Scholar 

  51. Cicchetti, D.V., Fleiss, J.L.: Comparison of the null distribution of weighted kappa and the C ordinal statistic. Appl. Psychol. Meas. 1, 195–201 (1977)

    Google Scholar 

  52. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)

    Google Scholar 

  53. Cohen, J.: Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychol. Bull. 70, 213–220 (1968)

    Google Scholar 

  54. Cohen, J.: Weighted chi square: An extension of the kappa method. Educ. Psychol. Meas. 32, 61–74 (1972)

    Google Scholar 

  55. Collins, M.F.: A permutation test for planar regression. Aust. J. Stat. 29, 303–308 (1987)

    Google Scholar 

  56. Corain, L., Salmaso, L.: A critical review and a comparative study on conditional permutation tests for two-way ANOVA. Commun. Stat. Simul. C 36, 791–805 (2007)

    MATH  MathSciNet  Google Scholar 

  57. Cormack, R.S., Mantel, N.: Fisher’s exact test: The marginal totals as seen from two different angles. Statistician 40, 27–34 (1991)

    Google Scholar 

  58. Cryan, M., Dyer, M.: A polynomial-time algorithm to approximately count contingency tables when the number of rows is constant. J. Comput. Syst. Sci. 67, 291–310 (2003)

    MATH  MathSciNet  Google Scholar 

  59. Curran-Everett, D.: Explorations in statistics: Standard deviations and standard errors. Adv. Physiol. Educ. 32, 203–208 (2008)

    Google Scholar 

  60. Curran-Everett, D.: Explorations in statistics: Regression. Adv. Physiol. Educ. 35, 347–352 (2011)

    Google Scholar 

  61. Curran-Everett, D.: Explorations in statistics: Permutation methods. Adv. Physiol. Educ. 36, 181–187 (2012)

    Google Scholar 

  62. Daniel, C.: Locating outliers in factorial experiments. Technometrics 2, 149–156 (1960)

    MATH  MathSciNet  Google Scholar 

  63. David, F.N.: Review of “Rank Correlation Methods” by M. G. Kendall. Biometrika 37, 190 (1950)

    Google Scholar 

  64. David, H.A.: The beginnings of randomization tests. Am. Stat. 62, 70–72 (2008)

    Google Scholar 

  65. Dodge, Y.: An introduction to statistical data analysis L 1-norm based. In: Dodge, Y. (ed.) Statistical Data Analysis Based on the L 1-norm and Related Methods, pp. 1–21. Elsevier, Amsterdam (1987) [Collection of invited papers presented at The First International Conference on Statistical Data Analysis Based on the L 1-norm and Related Methods, held in Neuchâtel, Switzerland, from 31 August to 4 September 1987]

    Google Scholar 

  66. Dodge, Y. (ed.): The Oxford Dictionary of Statistical Terms. Oxford University Press, Oxford (2003)

    MATH  Google Scholar 

  67. Draper, N.R., Stoneman, D.M.: Testing for the inclusion of variables in linear regression by a randomization technique. Technometrics 8, 695–699 (1966)

    MathSciNet  Google Scholar 

  68. Dwass, M.: Modified randomization tests for nonparametric hypotheses. Ann. Math. Stat. 28, 181–187 (1957)

    MATH  MathSciNet  Google Scholar 

  69. Edgeworth, F.Y.: The method of least squares. Philos. Mag. 5 16, 360–375 (1883)

    Google Scholar 

  70. Edgington, E.S.: Statistical Inference: The Distribution-free Approach. McGraw-Hill, New York (1969)

    Google Scholar 

  71. Edgington, E.S.: Randomization Tests, 2nd edn. Marcel Dekker, New York (1987)

    MATH  Google Scholar 

  72. Edgington, E.S., Khuller, P.L.V.: A randomization test computer program for trends in repeated-measures data. Educ. Psychol. Meas. 52, 93–95 (1992)

    Google Scholar 

  73. Edgington, E.S., Onghena, P.: Randomization Tests, 4th edn. Chapman & Hall/CRC, Boca Raton (2007)

    MATH  Google Scholar 

  74. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall/CRC, Boca Raton (1993)

    MATH  Google Scholar 

  75. Endler, J.A., Mielke, P.W.: Comparing entire colour patterns as birds see them. Biol. J. Linn. Soc. 86, 405–431 (2005)

    Google Scholar 

  76. Epstein, D.M., Dalinka, M.K., Kaplan, F.S., Aronchick, J.M., Marinelli, D.L., Kundel, H.L.: Observer variation in the detection of osteopenia. Skeletal Radiol. 15, 347–349 (1986)

    Google Scholar 

  77. Ernst, M.D.: Permutation methods: A basis for exact inference. Stat. Sci. 19, 676–685 (2004)

    MATH  MathSciNet  Google Scholar 

  78. Everitt, B.S.: Moments of the statistics kappa and weighted kappa. Br. J. Math. Stat. Psychol. 21, 97–103 (1968)

    Google Scholar 

  79. Feinstein, A.R.: Clinical Biostatistics XXIII: The role of randomization in sampling, testing, allocation, and credulous idolatry (Part 2). Clin. Pharmacol. Ther. 14, 898–915 (1973)

    Google Scholar 

  80. Finch, W.H., Davenport, T.: Performance of Monte Carlo permutation and approximate tests for multivariate means comparisons with small sample sizes when parametric assumptions are violated. Methodology 5, 60–70 (2009)

    Google Scholar 

  81. Finney, D.J.: Calibration guidelines challenge outlier practices. Am. Stat. 60, 309–314 (2006)

    MathSciNet  Google Scholar 

  82. Fisher, R.A.: On the interpretation of χ 2 from contingency tables, and the calculation of p. J. R. Stat. Soc. 85, 87–94 (1922)

    Google Scholar 

  83. Fisher, R.A.: Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh (1925)

    Google Scholar 

  84. Fisher, R.A.: The use of multiple measurements in taxonomic problems. Ann. Eugenic. 7, 179–188 (1936)

    Google Scholar 

  85. Fisher, R.A.: The Design of Experiments, 7th edn. Hafner, New York (1960)

    Google Scholar 

  86. Fitzmaurice, G.M., Lipsitz, S.R., Ibrahim, J.G.: A note on permutation tests for variance components in multilevel generalized linear mixed models. Biometrics 63, 942–946 (2007)

    MATH  MathSciNet  Google Scholar 

  87. Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley, New York (1981)

    MATH  Google Scholar 

  88. Fleiss, J.L., Cicchetti, D.V.: Inference about weighted kappa in the non-null case. Appl. Psychol. Meas. 2, 113–117 (1978)

    Google Scholar 

  89. Fleiss, J.L., Cohen, J., Everitt, B.S.: Large sample standard errors of kappa and weighted kappa. Psychol. Bull. 72, 323–327 (1969)

    Google Scholar 

  90. Fleiss, J.L., Levin, B., Paik, M.C.: Statistical Methods for Rates and Proportions, 5th edn. Wiley, New York (2003)

    MATH  Google Scholar 

  91. Fraker, M.E., Peacor, S.D.: Statistical tests for biological interactions: A comparison of permutation tests and analysis of variance. Acta Oecol. 33, 66–72 (2008)

    Google Scholar 

  92. Freedman, D., Lane, D.: A nonstochastic interpretation of reported significance levels. J. Bus. Econ. Stat. 1, 292–298 (1983)

    Google Scholar 

  93. Friedman, M.: The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Am. Stat. Assoc. 32, 675–701 (1937)

    Google Scholar 

  94. Fuechsle, M., Miwa, J.A., Mahapatra, S., Ryu, H., Lee, S., Warschkow, O., Hollenberg, L.C.L., Klimeck, G., Simmons, M.Y.: A single-atom transistor. Nat. Nanotechnol. http://www.nature.com/nnano/journal/vaop/ncurrent/full/nnano.2012.21.html (19 February 2012). Accessed 25 Feb 2012

  95. Gail, M.H., Tan, W.Y., Piantadosi, S.: Tests for no treatment effect in randomized clinical trials. Biometrika 75, 57–64 (1988)

    MATH  MathSciNet  Google Scholar 

  96. Geary, R.C.: Testing for normality. Biometrika 34, 209–242 (1947)

    MATH  MathSciNet  Google Scholar 

  97. Gebhard, J., Schmitz, N.: Permutation tests — a revival?! I. Optimum properties. Stat. Pap. 39, 75–85 (1998)

    MATH  MathSciNet  Google Scholar 

  98. Gibbons, J.D., Pratt, J.W.: P-values: Interpretation and methodology. Am. Stat. 29, 20–25 (1975)

    MATH  Google Scholar 

  99. Gill, P.M.W.: Efficient calculation of p-values in linear-statistic permutation significance tests. J. Stat. Comput. Simul. 77, 55–61 (2007)

    MATH  MathSciNet  Google Scholar 

  100. Gladwell, M.: Outliers: The Story of Success. Little, Brown, New York (2008)

    Google Scholar 

  101. Good, P.I.: Permutation, Parametric and Bootstrap Tests of Hypotheses, 2nd edn. Springer, New York (2000)

    Google Scholar 

  102. Good, P.I.: Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses, 2nd edn. Springer, New York (2000)

    Google Scholar 

  103. Good, P.I.: Resampling Methods: A Practical Guide to Data Analysis, 2nd edn. Birkhäuser, Boston (2001)

    Google Scholar 

  104. Good, P.I.: Efficiency comparisons of rank and permutation tests by Janice M. Weinberg and Stephen W. Lagakos in Statistics in Medicine 2001; 20:705–731. Stat. Med. 23, 857 (2004)

    Google Scholar 

  105. Good, P.I.: Efficiency comparisons of rank and permutation tests by Phillip I. Good in Statistics in Medicine 2004; 23:857. Stat. Med. 24, 1777–1781 (2005)

    Google Scholar 

  106. Good, P.I.: Permutation, Parametric and Bootstrap Tests of Hypotheses, 3rd edn. Springer, New York (2005)

    MATH  Google Scholar 

  107. Good, P.I.: Resampling Methods: A Practical Guide to Data Analysis, 3rd edn. Birkhäuser, Boston (2006)

    Google Scholar 

  108. Good, P.I., Xie, F.: Analysis of a crossover clinical trial by permutation methods. Contemp. Clin. Trials 29, 565–568 (2008)

    Google Scholar 

  109. Goodman, L.A., Kruskal, W.H.: Measures of association for cross classifications. J. Am. Stat. Assoc. 49, 732–764 (1954)

    MATH  Google Scholar 

  110. Graham, P., Jackson, R.: The analysis of ordinal agreement data: Beyond weighted kappa. J. Clin. Epidemiol. 46, 1055–1062 (1993)

    Google Scholar 

  111. Graves, T., Reese, C.S., Fitzgerald, M.: Hierarchical models for permutations: Analysis of auto racing results. J. Am. Stat. Assoc. 98, 282–291 (2003)

    MATH  MathSciNet  Google Scholar 

  112. Green, B.F.: Randomization tests. J. Am. Stat. Assoc. 76, 495 (1981) [Review of E.S. Edgington’s Randomization Tests by Bert F. Green]

    Google Scholar 

  113. Greenland, S.: On the logical justification of conditional tests for two-by-two contingency tables. Am. Stat. 45, 248–251 (1991)

    Google Scholar 

  114. Gumbel, E.J.: Discussion of the papers of Messrs. Anscombe and Daniel. Technometrics 2, 165–166 (1960)

    Google Scholar 

  115. Herman, P.G., Khan, A., Kallman, C.E., Rojas, K.A., Carmody, D.P., Bodenheimer, M.M.: Limited correlation of left ventricular end-diastolic pressure with radiographic assessment of pulmonary hemodynamics. Radiology 174, 721–724 (1990)

    Google Scholar 

  116. Higgins, J.J., Blair, R.C.: Comment on “Why permutation tests are superior to t and F tests in biomedical research” by J. Ludbrook and H.A.F. Dudley. Am. Stat. 54, 86 (2000)

    Google Scholar 

  117. Hilbert, M.: How much information is there in the “information society?”. Significance 9, 8–12 (2012)

    Google Scholar 

  118. Hirji, K.F.: Exact Analysis of Discrete Data. Chapman & Hall/CRC, Boca Raton (2006)

    MATH  Google Scholar 

  119. Hirji, K.F., Tan, S., Elashoff, R.M.: A quasi-exact test for comparing two binomial proportions. Stat. Med. 10, 1137–1153 (1991)

    Google Scholar 

  120. Hitchcock, D.B.: Yates and contingency tables: 75 years later. Elec. J. Hist. Prob. Stat. 5, 1–14 (2009)

    MATH  MathSciNet  Google Scholar 

  121. Hodges, J.L., Lehmann, E.L.: The efficiency of some non-parametric competitors of the t-test. Ann. Math. Stat. 27, 324–335 (1956)

    MATH  MathSciNet  Google Scholar 

  122. Holley, J.W., Guilford, J.P.: A note on the g index of agreement. Educ. Psychol. Meas. 4, 749–753 (1964)

    Google Scholar 

  123. Horn, S.D.: Goodness-of-fit tests for discrete data: A review and an application to a health impairment scale. Biometrics 33, 237–247 (1977)

    MATH  MathSciNet  Google Scholar 

  124. Hotelling, H., Pabst, M.R.: Rank correlation and tests of significance involving no assumption of normality. Ann. Math. Stat. 7, 29–43 (1936)

    Google Scholar 

  125. Howard, J.V.: The 2 × 2 table: A discussion from a Bayesian viewpoint. Stat. Sci. 13, 351–367 (1998)

    MATH  Google Scholar 

  126. Huang, A., Jin, R., Robinson, J.: Robust permutation tests for two samples. J. Stat. Plan. Infer. 139, 2631–2642 (2009)

    MATH  MathSciNet  Google Scholar 

  127. Huber, P.J.: Robust estimation of a location parameter. Ann. Math. Stat. 35, 73–101 (1964)

    MATH  Google Scholar 

  128. Hubert, L.J.: Kappa revisited. Psychol. Bull. 84, 289–297 (1977)

    Google Scholar 

  129. Hubert, L.J.: A general formula for the variance of Cohen’s weighted kappa. Psychol. Bull. 85, 183–184 (1978)

    Google Scholar 

  130. Huh, M.H., Jhun, M.: Random permutation testing in multiple linear regression. Commun. Stat. Theor. M. 30, 2023–2032 (2001)

    MATH  MathSciNet  Google Scholar 

  131. Irwin, J.O.: Tests of significance for differences between percentages based on small numbers. Metron 12, 83–94 (1935)

    Google Scholar 

  132. Janson, S., Vegelius, J.: On generalizations of the g index and the phi coefficient to nominal scales. Multivar. Behav. Res. 14, 255–269 (1979)

    Google Scholar 

  133. Janssen, A., Pauls, T.: How do bootstrap and permutation tests work? Ann. Stat. 31, 768–806 (2003)

    MATH  MathSciNet  Google Scholar 

  134. Jiang, W., Kalbfleisch, J.D.: Permutation methods in relative risk regression models. J. Stat. Plan. Infer. 138, 416–431 (2008)

    MATH  MathSciNet  Google Scholar 

  135. Jin, R., Robinson, J.: Robust permutation tests for one sample. J. Stat. Plan. Infer. 116, 475–487 (2003)

    MATH  MathSciNet  Google Scholar 

  136. Johnston, J.E., Berry, K.J., Mielke, P.W.: Permutation tests: Precision in estimating probability values. Percept. Motor Skill. 105, 915–920 (2007)

    Google Scholar 

  137. Jolayemi, E.T.: On the measure of agreement between two raters. Biometrical J. 32, 87–93 (1990)

    Google Scholar 

  138. Jung, B.C., Jhun, M., Song, S.H.: A new random permutation test in ANOVA models. Stat. Pap. 48, 47–62 (2007)

    MathSciNet  Google Scholar 

  139. Kaiser, J.: An exact and a Monte Carlo proposal to the Fisher–Pitman permutation tests for paired replicates and for independent samples. Stata J. 7, 402–412 (2007)

    Google Scholar 

  140. Kaufman, E.H., Taylor, G.D., Mielke, P.W., Berry, K.J.: An algorithm and FORTRAN program for multivariate LAD (ℓ 1 of ℓ 2) regression. Computing 68, 275–287 (2002)

    MATH  MathSciNet  Google Scholar 

  141. Keller-McNulty, S., Higgins, J.J.: Effect of tail weight and outliers and power and type-I error of robust permutation tests for location. Commun. Stat. Simul. C 16, 17–35 (1987)

    MathSciNet  Google Scholar 

  142. Kempthorne, O.: The randomization theory of experimental inference. J. Am. Stat. Assoc. 50, 946–967 (1955)

    MathSciNet  Google Scholar 

  143. Kendall, M.G.: A new measure of rank correlation. Biometrika 30, 81–93 (1938)

    MATH  MathSciNet  Google Scholar 

  144. Kendall, M.G.: Rank Correlation Methods. Griffin, London (1948)

    MATH  Google Scholar 

  145. Kendall, M.G.: Studies in the history of probability and statistics: XI. Daniel Bernoulli on maximum likelihood. Biometrika 48, 1–18 (1961)

    MATH  MathSciNet  Google Scholar 

  146. Kendall, M.G.: Rank Correlation Methods, 3rd edn. Griffin, London (1962)

    Google Scholar 

  147. Kendall, M.G., Babington Smith, B.: On the method of paired comparisons. Biometrika 31, 324–345 (1940)

    MATH  MathSciNet  Google Scholar 

  148. Kennedy, P.E.: Randomization tests in econometrics. J. Bus. Econ. Stat. 13, 85–94 (1995)

    Google Scholar 

  149. Kennedy, P.E., Cade, B.S.: Randomization tests for multiple regression. Commun. Stat. Simul. C 25, 923–936 (1996)

    MATH  Google Scholar 

  150. Kim, M.J., Nelson, C.R., Startz, R.: Mean revision in stock prices? A reappraisal of the empirical evidence. Rev. Econ. Stud. 58, 515–528 (1991)

    Google Scholar 

  151. Kingman, A.: Beyond weighted kappa when evaluating examiner agreement for ordinal responses. J. Dent. Res. 81, A219 (2002)

    Google Scholar 

  152. Knijnenburg, T.A., Wessels, L.F.A., Reinders, M.J.T., Shmulevich, I.: Fewer permutations, more accurate P-values. Intell. Syst. Mol. Biol. 25, i161–i168 (2009)

    Google Scholar 

  153. Kramer, M.S., Feinstein, A.R.: Clinical biostatistics: LIV. The biostatistics of concordance. Clin. Pharm. Therap. 29, 111–123 (1981)

    Google Scholar 

  154. Kruskal, W.H.: Discussion of the papers of Messrs. Anscombe and Daniel. Technometrics 2, 157–158 (1960)

    MathSciNet  Google Scholar 

  155. Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47, 583–621 (1952) [Erratum: J. Am. Stat. Assoc. 48, 907–911 (1953)]

    Google Scholar 

  156. Kundel, H.L., Polansky, M.: Measurement of observer agreement. Radiology 228, 303–308 (2003)

    Google Scholar 

  157. LaFleur, B.J., Greevy, R.A.: Introduction to permutation and resampling-based hypothesis tests. J. Clin. Child Adolesc. 38, 286–294 (2009)

    Google Scholar 

  158. Lahiri, S.N.: Resampling Methods for Dependent Data. Springer, New York (2003)

    MATH  Google Scholar 

  159. Lancaster, H.O.: The combination of probabilities arising from data in discrete distributions. Biometrika 36, 370–382 (1949)

    MathSciNet  Google Scholar 

  160. Lancaster, H.O.: Significance test in discrete distributions. J. Am. Stat. Assoc. 56, 223–234 (1961) [Corrigendum: J. Am. Stat. Assoc. 57, 919 (1962)]

    Google Scholar 

  161. Lance, C.E.: More statistical and methodological myths and urban legends. Organ. Res. Methods 14, 279–286 (2011)

    MATH  Google Scholar 

  162. Landis, J.R., Koch, G.G.: The measurement of observer agreement for ordinal data. Biometrics 33, 159–174 (1977)

    MATH  MathSciNet  Google Scholar 

  163. Legendre, P.: Species associations: The Kendall coefficient of concordance revisited. J. Agric. Biol. Environ. Sci. 10, 226–245 (2005)

    Google Scholar 

  164. Legendre, P., Gallagher, E.D.: Ecologically meaningful transformations for ordination of species data. Oecologia 129, 271–280 (2001)

    Google Scholar 

  165. Lehmann, E.L.: Parametrics vs. nonparametrics: Two alternative methodologies. J. Nonparametr. Stat. 21, 397–405 (2009)

    Google Scholar 

  166. Lehmann, E.L.: Fisher, Neyman, and the Creation of Classical Statistics. Springer, New York (2011)

    MATH  Google Scholar 

  167. Littell, R.C., Folks, J.L.: Asymptotic optimality of Fisher’s method of combining independent tests. J. Am. Stat. Assoc. 66, 802–806 (1971)

    MATH  MathSciNet  Google Scholar 

  168. Littell, R.C., Folks, J.L.: Asymptotic optimality of Fisher’s method of combining independent tests: II. J. Am. Stat. Assoc. 68, 193–194 (1973)

    MATH  MathSciNet  Google Scholar 

  169. Little, R.J.A.: Testing the equality of two independent binomial proportions. Am. Stat. 43, 283–288 (1989)

    Google Scholar 

  170. Lomb, N.: Transit of Venus: 1631 to the Present. Powerhouse Museum, Sydney (2011)

    Google Scholar 

  171. Long, M.A., Berry, K.J., Mielke, P.W.: A note on tests of significance for multiple regression coefficients. Psychol. Rep. 100, 339–345 (2007)

    Google Scholar 

  172. Long, M.A., Berry, K.J., Mielke, P.W.: Tetrachoric correlation: A permutation alternative. Educ. Psychol. Meas. 69, 429–437 (2009)

    MathSciNet  Google Scholar 

  173. Ludbrook, J.: Statistical techniques for comparing measures and methods of measurement: A critical review. Clin. Exp. Pharmacol. Physiol. 29, 527–536 (2002)

    Google Scholar 

  174. Ludbrook, J.: Outlying observations and missing values: How should they be handled? Clin. Exp. Pharmacol. Physiol. 35, 670–678 (2008)

    Google Scholar 

  175. Ludbrook, J., Dudley, H.A.F.: Why permutation tests are superior to t and F tests in biomedical research. Am. Stat. 52, 127–132 (1998)

    Google Scholar 

  176. Ludbrook, J., Dudley, H.A.F.: Discussion of “Why permutation tests are superior to t and F tests in biomedical research” by J. Ludbrook and H.A.F. Dudley. Am. Stat. 54, 87 (2000)

    Google Scholar 

  177. Lunneborg, C.E.: Data Analysis by Resampling: Concepts and Applications. Duxbury, Pacific Grove (2000)

    Google Scholar 

  178. Lyons, D.: In race for fastest computer, China outpaces U.S. Newsweek 158, 57–59 (5 December 2011)

    Google Scholar 

  179. Maclure, M., Willett, W.C.: Misinterpretation and misuse of the kappa statistic. Am. J. Epidemiol. 126, 161–169 (1987)

    Google Scholar 

  180. Maltz, M.D.: Deviating from the mean: The declining significance of significance. J. Res. Crime Delinq. 31, 434–463 (1994)

    Google Scholar 

  181. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology. Chapman & Hall, London (1991)

    MATH  Google Scholar 

  182. Manly, B.F.J.: Randomization and Monte Carlo Methods in Biology, 2nd edn. Chapman & Hall, London (1997)

    MATH  Google Scholar 

  183. Manly, B.F.J.: Randomization, Bootstrap and Monte Carlo Methods in Biology, 3rd edn. Chapman & Hall/CRC, Boca Raton (2007)

    MATH  Google Scholar 

  184. Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 50–60 (1947)

    MATH  MathSciNet  Google Scholar 

  185. Martín Andrés, A.: Comments on “Chi-squared and Fisher–Irwin tests of two-by-two tables with small sample recommendations” by I. Campbell. Stat. Med. 27, 1791–1795 (2008)

    Google Scholar 

  186. Martín Andrés, A., Herranz Tejedor, I.H.: Is Fisher’s exact test very conservative? Comput. Stat. Data Anal. 19, 579–591 (1995)

    MATH  Google Scholar 

  187. Martín Andrés, A., Sánchez Quevedo, M.J., Tapia García, J.M., Silva Mato, A.: On the validity condition of the chi-squared test in 2 × 2 tables. Test 14, 1–30 (2005)

    MathSciNet  Google Scholar 

  188. Maxwell, A.E.: Coefficients of agreement between observers and their interpretation. Br. J. Psychiatr. 130, 79–83 (1977)

    Google Scholar 

  189. May, R.B., Hunter, M.A.: Some advantages of permutation tests. Can. Psychol. 34, 401–407 (1993)

    Google Scholar 

  190. McQueen, G.: Long-horizon mean-reverting stock priced revisited. J. Financ. Quant. Anal. 27, 1–17 (1992)

    Google Scholar 

  191. Mewhort, D.J.K., Johns, B.T., Kelly, M.: Applying the permutation test to factorial designs. Behav. Res. Methods 42, 366–372 (2010)

    Google Scholar 

  192. Micceri, T.: The unicorn, the normal curve, and other improbable creatures. Psychol. Bull. 105, 156–166 (1989)

    Google Scholar 

  193. Mielke, P.W.: Meteorological applications of permutation techniques based on distance functions. In: Krishnaiah, P.R., Sen, P.K. (eds.) Handbook of Statistics, vol. IV, pp. 813–830. North-Holland, Amsterdam (1984)

    Google Scholar 

  194. Mielke, P.W.: Geometric concerns pertaining to applications of statistical tests in the atmospheric sciences. J. Atmos. Sci. 42, 1209–1212 (1985)

    Google Scholar 

  195. Mielke, P.W.: Non-metric statistical analyses: Some metric alternatives. J. Stat. Plan. Infer. 13, 377–387 (1986)

    MATH  MathSciNet  Google Scholar 

  196. Mielke, P.W.: L 1, L 2 and L ∞ regression models: Is there a difference? J. Stat. Plan. Infer. 16, 430 (1987)

    Google Scholar 

  197. Mielke, P.W., Berry, K.J.: Non-asymptotic inferences based on the chi-square statistic for r by c contingency tables. J. Stat. Plan. Infer. 12, 41–45 (1985)

    MathSciNet  Google Scholar 

  198. Mielke, P.W., Berry, K.J.: Cumulant methods for analyzing independence of r-way contingency tables and goodness-of-fit frequency data. Biometrika 75, 790–793 (1988)

    MATH  MathSciNet  Google Scholar 

  199. Mielke, P.W., Berry, K.J.: Euclidean distance based permutation methods in atmospheric science. Data Min. Knowl. Disc. 4, 7–27 (2000)

    MATH  Google Scholar 

  200. Mielke, P.W., Berry, K.J.: Permutation Methods: A Distance Function Approach. Springer, New York (2001)

    Google Scholar 

  201. Mielke, P.W., Berry, K.J.: Categorical independence tests for large sparse R-way contingency tables. Percept. Motor Skill. 95, 606–610 (2002)

    Google Scholar 

  202. Mielke, P.W., Berry, K.J.: Multivariate multiple regression analyses: A permutation method for linear models. Psychol. Rep. 91, 3–9 (2002)

    Google Scholar 

  203. Mielke, P.W., Berry, K.J.: Multivariate multiple regression prediction models: A Euclidean distance approach. Psychol. Rep. 92, 763–769 (2003)

    Google Scholar 

  204. Mielke, P.W., Berry, K.J.: Permutation Methods: A Distance Function Approach, 2nd edn. Springer, New York (2007)

    Google Scholar 

  205. Mielke, P.W., Berry, K.J.: Two-sample multivariate similarity permutation comparison. Psychol. Rep. 100, 257–262 (2007)

    Google Scholar 

  206. Mielke, P.W., Berry, K.J.: A note on Cohen’s weighted kappa coefficient of agreement with linear weights. Stat. Methodol. 6, 439–446 (2009)

    MathSciNet  Google Scholar 

  207. Mielke, P.W., Berry, K.J., Johnston, J.E.: Comparisons of continuous and discrete methods for combining probability values associated with matched-pairs t-test data. Percept. Motor Skill. 100, 799–805 (2005)

    Google Scholar 

  208. Mielke, P.W., Berry, K.J., Johnston, J.E.: A FORTRAN program for computing the exact variance of weighted kappa. Percept. Motor Skill. 101, 468–472 (2005)

    Google Scholar 

  209. Mielke, P.W., Berry, K.J., Johnston, J.E.: The exact variance of weighted kappa with multiple raters. Psychol. Rep. 101, 655–660 (2007)

    Google Scholar 

  210. Mielke, P.W., Berry, K.J., Johnston, J.E.: Resampling programs for multiway contingency tables with fixed marginal frequency totals. Psychol. Rep. 101, 18–24 (2007)

    Google Scholar 

  211. Mielke, P.W., Berry, K.J., Johnston, J.E.: Resampling probability values for weighted kappa with multiple raters. Psychol. Rep. 102, 606–613 (2008)

    Google Scholar 

  212. Mielke, P.W., Berry, K.J., Johnston, J.E.: Unweighted and weighted kappa as measures of agreement for multiple judges. Int. J. Manag. 26, 213–223 (2009)

    Google Scholar 

  213. Mielke, P.W., Berry, K.J., Johnston, J.E.: Robustness without rank order statistics. J. Appl. Stat. 38, 207–214 (2011)

    MathSciNet  Google Scholar 

  214. Mielke, P.W., Berry, K.J., Medina, J.G.: Climax I and II: Distortion resistant residual analyses. J. Appl. Meterol. 21, 788–792 (1982)

    Google Scholar 

  215. Mielke, P.W., Johnston, J.E., Berry, K.J.: Combining probability values from independent permutation tests: A discrete analog of Fisher’s classical method. Psychol. Rep. 95, 449–458 (2004)

    Google Scholar 

  216. Mielke, P.W., Long, M.A., Berry, K.J., Johnston, J.E.: g-treatment ridit analysis: Resampling permutation methods. Stat. Methodol. 6, 223–229 (2009)

    Google Scholar 

  217. Mood, A.M.: On the asymptotic efficiency of certain nonparametric two-sample tests. Ann. Math. Stat. 25, 514–522 (1954)

    MATH  MathSciNet  Google Scholar 

  218. Mordkoff, J.T.: The assumption(s) of normality. http://www2.psychology.uiowa.edu/faculty/mordkoff/GradStats/part%20I/I.07%20normal.pdf (2011). Accessed 18 Aug 2013

  219. Murphy, K.R., Cleveland, J.: Understanding Performance Appraisal: Social, Organizational, and Goal-based Perspectives. Sage, Thousand Oaks (1995)

    Google Scholar 

  220. Newcomb, S.: Researches on the motion of the moon, Part II. The mean motion of the moon and other astronomical elements derived from observations of eclipses and occultations extending from the period of the Babylonians until a.d. 1908. Astron. Pap. 9, 1–249 (1912)

    Google Scholar 

  221. Noreen, E.W.: Computer-intensive Methods For Testing Hypotheses: An Introduction. Wiley, New York (1989)

    Google Scholar 

  222. Norman, R.G., Scott, M.A.: Measurement of inter-rater agreement for transient events using Monte Carlo sampled permutations. Stat. Med. 26, 931–942 (2007)

    MathSciNet  Google Scholar 

  223. O’Boyle, Jr., E., Aguinis, H.: The best and the rest: Revisiting the norm of normality of individual performance. Pers. Psychol. 65, 79–119 (2012)

    Google Scholar 

  224. O’Gorman, T.W.: The performance of randomization tests that use permutations of independent variables. Commun. Stat. Simul. C 34, 895–908 (2005)

    MATH  MathSciNet  Google Scholar 

  225. Oja, H.: On permutation tests in multiple regression and analysis of covariance problems. Aust. J. Stat. 29, 91–100 (1987)

    MATH  MathSciNet  Google Scholar 

  226. Önder, H.: Using permutation tests to reduce type I and II errors for small ruminant research. J. Appl. Anim. Res. 32, 69–72 (2007)

    Google Scholar 

  227. Önder, H.: A comparative study of permutation tests with Euclidean and Bray–Curtis distances for common agricultural distributions in regression. J. Appl. Anim. Res. 34, 133–136 (2008)

    Google Scholar 

  228. Pareto, V.F.D.: L’economie et la sociologie au point de vue scientifique (Economics and sociology from a scientific perspective). In: Écrites Sociologiques Mineurs, vol. 22 of Oeuvres Complètes. Droz, Geneva (1980)

    Google Scholar 

  229. Patefield, W.M.: Algorithm 159: An efficient method of generating random r × c tables with given row and column totals. J. R. Stat. Soc. C Appl. 30, 91–97 (1981)

    MATH  Google Scholar 

  230. Pearson, E.S.: The choice of statistical tests illustrated on the interpretation of data classed in a 2 × 2 table. Biometrika 34, 139–167 (1947)

    MATH  MathSciNet  Google Scholar 

  231. Pearson, E.S.: On questions raised by the combination of tests based on discontinuous distributions. Biometrika 37, 383–398 (1950)

    MATH  MathSciNet  Google Scholar 

  232. Pearson, K.: On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philos. Mag. 5 50, 157–175 (1900)

    Google Scholar 

  233. Pearson, K.: On the laws of inheritance in man: II. On the inheritance of the mental and moral characters in man, and its comparison with the inheritance of the physical characters. Biometrika 3, 131–190 (1904)

    Google Scholar 

  234. Pearson, K.: Mathematical contributions to the theory of evolution, XVI. On further methods of determining correlation. In: Drapers’ Company Research Memoirs, Biometric Series IV, pp. 1–39. Dulau and Company, London (1907)

    Google Scholar 

  235. Perkins, S.M., Becker, M.P.: Assessing rater agreement using marginal association models. Stat. Med. 21, 1743–1760 (2002)

    Google Scholar 

  236. Pesarin, F.: Multivariate Permutation Tests: With Applications in Biostatistics. Wiley, Chichester (2001)

    Google Scholar 

  237. Pesarin, F., Salmaso, L.: Exact permutation tests for unreplicated factorials. Appl. Stoch. Model. Bus. 18, 287–299 (2002)

    MATH  MathSciNet  Google Scholar 

  238. Pesarin, F., Salmaso, L.: Permutation Tests for Complex Data: Theory, Applications and Software. Wiley, Chichester (2010)

    Google Scholar 

  239. Raab, G.M., Butcher, I.: Randomization inference for balanced cluster-randomized trials. Clin. Trials 2, 130–140 (2005)

    Google Scholar 

  240. Radlow, R., Alf, Jr., E.F.: An alternate multinomial assessment of the accuracy of the χ 2 test of goodness of fit. J. Am. Stat. Assoc. 70, 811–813 (1975)

    MATH  Google Scholar 

  241. Ralph, N.: Processors: What to expect from CPUs in 2012. http://www.pcworld.com/article/246688/processors_what_to_expect_from_cpus_in_2012.html (27 December 2011). Accessed 29 Apr 2012

  242. Reid, N.: The roles of conditioning in inference. Stat. Sci. 10, 138–157 (1995) [See also the accompanying discussion in the same issue on pages 173–199]

    Google Scholar 

  243. Reiss, P.T., Stevens, M.H.H., Shehzad, Z., Petkova, E., Milham, M.P.: On distance-based permutation tests for between-group comparisons. Biometrics 66, 636–643 (2010)

    MATH  MathSciNet  Google Scholar 

  244. Richardson, J.T.E.: Variants of chi-square for 2 × 2 contingency tables. Br. J. Math. Stat. Psychol. 43, 309–326 (1990)

    Google Scholar 

  245. Richardson, J.T.E.: The analysis of 2 × 1 and 2 × 2 contingency tables: A historical review. Stat. Methods Med. Res. 3, 107–134 (1994)

    Google Scholar 

  246. Routledge, R.D.: Resolving the conflict over Fisher’s exact test. Can. J. Stat. 20, 201–209 (1992)

    MATH  MathSciNet  Google Scholar 

  247. Roy, T.: The effect of heteroscedasticity and outliers on the permutation t-test. J. Stat. Comput. Simul. 72, 23–26 (2002)

    Google Scholar 

  248. Saal, F.E., Downey, R.G., Lahey, M.A.: Rating the ratings: Assessing the quality of rating data. Psychol. Bull. 88, 413–428 (1980)

    Google Scholar 

  249. Sakaori, F.: Permutation test for equality of correlation coefficients in two populations. Commun. Stat. Simul. C 31, 641–651 (2002)

    MATH  MathSciNet  Google Scholar 

  250. Schmidt, F.L., Johnson, R.H.: Effect of race on peer ratings in an industrial situation. J. Appl. Psychol. 57, 237–241 (1973)

    Google Scholar 

  251. Schouten, H.J.A.: Measuring pairwise agreement among many observers. Biometrical J. 22, 497–504 (1980)

    MATH  MathSciNet  Google Scholar 

  252. Schouten, H.J.A.: Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Stat. Neerl. 36, 45–61 (1982)

    MATH  Google Scholar 

  253. Schouten, H.J.A., Molenaar, I.W., van Strik, R., Boomsma, A.: Comparing two independent binomial proportions by a modified chi square test. Biometrical J. 22, 241–248 (1980)

    MATH  Google Scholar 

  254. Schuster, C.: A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educ. Psychol. Meas. 64, 243–253 (2004)

    MathSciNet  Google Scholar 

  255. Schuster, C., Smith, D.A.: Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients. Psychometrika 70, 135–146 (2005)

    MathSciNet  Google Scholar 

  256. Shah, A.: Intel unveils new core processors code-named Ivy Bridge. http://www.itworld.com/hardware/270726/intel-unveils-new-core-processors-code-named-ivy-bridge (23 April 2012). Accessed 29 Apr 2012

  257. Sheehan, W., Westfall, J.: The Transits of Venus. Prometheus, Amherst (2004)

    Google Scholar 

  258. Shepherd, J.: World education rankings: Which country does best at reading, maths and science?. http://www.guardian.co.uk/news/datablog/2010/dec/07/world-education-rankings-maths-science-reading (2010). Accessed 16 Feb 2012

  259. Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures, 3rd edn. Chapman & Hall/CRC, Boca Raton (2004)

    MATH  Google Scholar 

  260. Short, J.: An account of the transit of Venus over the Sun, on Saturday morning, 6th June 1761, at Savile-House, about 8′ ′ of time west of St. Paul’s, London. Philos. Trans. R. Soc. Lond. 52, 178–182 (1761–1762) [Published in the Philosophical Transactions of the Royal Society of London (1683–1775)]

    Google Scholar 

  261. Short, J.: The observations of the internal contact of Venus with the Sun’s limb, in the late transit, made in different places of Europe, compared with the time of the same contact observed at the Cape of Good Hope, and the parallax of the Sun from thence determined. By James Short, A.M. F.R.S. Philos. Trans. R. Soc. Lond. 52, 611–628 (1761–1762) [Published in the Philosophical Transactions of the Royal Society of London (1683–1775)]

    Google Scholar 

  262. Siegfried, T.: Odds are, it’s wrong. Sci. News 177, 26–29 (27 March 2010)

    Google Scholar 

  263. Smalley, E.: Ultimate Computing. Discover, pp. 10–11 (July/August 2011)

    Google Scholar 

  264. Somers, R.H.: A new asymmetric measure of association for ordinal variables. Am. Sociol. Rev. 27, 799–811 (1962)

    Google Scholar 

  265. Sparkes, M.: MareNostrum, the world’s most gorgeous super-computer. http://gizmodo.com/293608/marenostrum-the-worlds-most-gorgeous-super+computer (2007). Accessed 12 Mar 2012

  266. Spearman, C.E.: The proof and measurement of association between two things. Am. J. Psychol. 15, 72–101 (1904)

    Google Scholar 

  267. Spearman, C.E.: ‘Footrule’ for measuring correlation. Br. J. Psychol. 2, 89–108 (1906)

    Google Scholar 

  268. Spearman, C.E.: Correlation calculated from faulty data. Br. J. Psychol. 3, 271–295 (1910)

    Google Scholar 

  269. Still, A.W., White, A.P.: The approximate randomization test as an alternative to the F test in analysis of variance. Br. J. Math. Stat. Psychol. 34, 243–252 (1981)

    Google Scholar 

  270. Stone, E.G.: On the rejection of discordant observations. Mon. Not. R. Astron. Soc. 34, 9–15 (1873)

    MATH  Google Scholar 

  271. Stuart, A.: The estimation and comparison of strengths of association in contingency tables. Biometrika 40, 105–110 (1953)

    MATH  MathSciNet  Google Scholar 

  272. Stuart, A., Ord, J.K., Arnold, S.: Kendall’s Advanced Theory of Statistics, vol. 2A, 6th edn. Arnold, London (1999)

    Google Scholar 

  273. Suissa, S., Shuster, J.J.: Exact unconditional sample sizes for the 2 by 2 binomial trial. J. R. Stat. Soc. A Gen. 148, 317–327 (1985)

    MATH  MathSciNet  Google Scholar 

  274. Suissa, S., Shuster, J.J.: The 2 × 2 matched-pairs trial: Exact unconditional design and analysis. Biometrics 47, 361–372 (1991)

    MathSciNet  Google Scholar 

  275. Taha, M.A.H.: Rank test for scale parameter for asymmetrical one-sided distributions. Publ. Inst. Stat. Paris 13, 169–180 (1964)

    MATH  MathSciNet  Google Scholar 

  276. Taplin, S.H., Rutter, C.M., Elmore, J.G., Seger, D., White, D., Brenner, R.J.: Accuracy of screening mammography using single versus independent double interpretation. Am. J. Roentgenol. 174, 1257–1262 (2000)

    Google Scholar 

  277. ter Braak, C.J.F.: Permutation versus bootstrap significance tests in multiple regression and ANOVA. In: Jöckel, K.H., Rothe, G., Sendler, W. (eds.) Bootstrapping and Related Techniques, pp. 79–86. Springer, Berlin (1992)

    Google Scholar 

  278. Thompson, D.W.: On Growth and Form: The Complete Revised Edition. Dover, New York (1992)

    Google Scholar 

  279. TOP500 Supercomputing Sites. http://www.top500.org (2011). Accessed 12 Mar 2012

  280. Toppo, G.: Study’s rankings boost U.S. schools. USA Today. http://www.usatoday.com/news/education/story/2012-02-16/us-schools-global-ranking/53110494/1 (16 February 2012). Accessed 17 Feb 2012

  281. Tukey, J.W.: Discussion of the papers of Messrs. Anscombe and Daniel. Technometrics 2, 160–165 (1960)

    MathSciNet  Google Scholar 

  282. Tukey, J.W.: Randomization and re-randomization: The wave of the past in the future. In: Statistics in the Pharmaceutical Industry: Past, Present and Future. Philadelphia Chapter of the American Statistical Association (June 1988) [Presented at a Symposium in Honor of Joseph L. Ciminera held in June 1988 at Philadelphia, Pennsylvania]

    Google Scholar 

  283. Upton, G.J.G.: A comparison of alternative tests for the 2 × 2 comparative trial. J. R. Stat. Soc. A Gen. 145, 86–105 (1982)

    MathSciNet  Google Scholar 

  284. Upton, G.J.G.: Fisher’s exact test. J. R. Stat. Soc. A Gen. 155, 395–402 (1992)

    Google Scholar 

  285. van den Brink, W.P., van den Brink, S.G.L.: A comparison of the power of the t test, Wilcoxon’s test, and the approximate permutation test for the two-sample location problem. Br. J. Math. Stat. Psychol. 42, 183–189 (1989)

    MATH  Google Scholar 

  286. Vanbelle, S., Albert, A.: A note on the linearly weighted kappa coefficient for ordinal scales. Stat. Methodol. 6, 157–163 (2008)

    MathSciNet  Google Scholar 

  287. Vuong, A.: A new chip off the old block. Denver Post 120, 1A, 16A (2 June 2013)

    Google Scholar 

  288. Wald, A., Wolfowitz, J.: An exact test for randomness in the non-parametric case based on serial correlation. Ann. Math. Stat. 14, 378–388 (1943)

    MATH  MathSciNet  Google Scholar 

  289. Weber, B., Mahapatra, S., Ryu, H., Fuhrer, A., Reusch, C.G., Thompson, D.L., Lee, W.C.T., Klimeck, G., Hollenberg, L.C.L., Simmons, M.Y.: Ohm’s law survives to the atomic scale. Science 335, 64–67 (6 January 2012)

    Google Scholar 

  290. Weinberg, J.M., Lagakos, S.W.: Efficiency comparisons of rank and permutation tests based on summary statistics computed from repeated measures data. Stat. Med. 20, 705–731 (2001)

    Google Scholar 

  291. Welch, W.J.: Construction of permutation tests. J. Am. Stat. Assoc. 85, 693–698 (1990)

    Google Scholar 

  292. Westlund, K.B., Kurland, L.T.: Studies on multiple sclerosis in Winnipeg, Manitoba and New Orleans, Louisiana. Am. J. Hyg. 57, 380–396 (1953)

    Google Scholar 

  293. Wheldon, M.C., Anderson, M.J., Johnson, B.W.: Identifying treatment effects in multi-channel measurements in electroencephalographic studies: Multivariate permutation tests and multiple comparisons. Aust. N. Z. J. Stat. 49, 397–413 (2007)

    MATH  MathSciNet  Google Scholar 

  294. Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1, 80–83 (1945)

    Google Scholar 

  295. Wong, R.K.W., Chidambaram, N., Mielke, P.W.: Application of multi-response permutation procedures and median regression for covariate analyses of possible weather modification effects on hail responses. Atmos. Ocean 21, 1–13 (1983)

    Google Scholar 

  296. Yamada, T., Sugiyama, T.: On the permutation test in canonical correlation analysis. Comput. Stat. Data Anal. 50, 2111–2123 (2006)

    MATH  MathSciNet  Google Scholar 

  297. Yates, F.: Contingency tables involving small numbers and the χ 2 test. Suppl. J. R. Stat. Soc. 1, 217–235 (1934)

    MATH  Google Scholar 

  298. Yates, F.: Tests of significance for 2 × 2 contingency tables (with discussion). J. R. Stat. Soc. A Gen. 147, 426–463 (1984)

    MATH  MathSciNet  Google Scholar 

  299. Yu, J., Kepner, J.L., Iyer, R.: Exact tests using two correlated binomial variables in contemporary cancer clinical trials. Biometrical J. 51, 899–914 (2009)

    MathSciNet  Google Scholar 

  300. Zhang, S.: The split sample permutation t-tests. J. Stat. Plan. Infer. 139, 3512–3524 (2009)

    MATH  Google Scholar 

  301. Zwick, R.: Another look at interrater agreement. Psychol. Bull. 103, 374–378 (1988)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Sociology, Colorado State University, Fort Collins, CO, USA

    Kenneth J. Berry

  2. U.S. Government, Alexandria, VA, USA

    Janis E. Johnston

  3. Department of Statistics, Colorado State University, Fort Collins, CO, USA

    Paul W. Mielke Jr.

Authors
  1. Kenneth J. Berry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janis E. Johnston
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul W. Mielke Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Berry, K.J., Johnston, J.E., Mielke, P.W. (2014). Beyond 2000. In: A Chronicle of Permutation Statistical Methods. Springer, Cham. https://doi.org/10.1007/978-3-319-02744-9_6

Download citation

Publish with us

Policies and ethics

Search

Navigation