Abstract
The statistical analysis of the data obtained in an experiment is an elementary part of an experimental investigation. It makes it possible both to interpret the results of an experiment in an appropriate way and to support the experimental examination of the research question. It also allows the experimental setup to be improved before the actual experiment commences. Our main objective is to develop a broad guide to the use of statistical methods that systematizes and presents the content of the most important classes of methods and identifies the most important prerequisites for their application.
To call in the statistician after the experiment is done may be no more than asking him to perform a postmortem examination … he may be able to say what the experiment died of.
R.A. Fisher, Indian Statistical Congress, Sankhya, 1938
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Treatment variables are usually factor variables, i.e. variables that can only take a limited number of values. The values of a factor variable are generally referred to as levels.
- 2.
We will discuss a specific block design later in 7 Sect. 4.3.2.
- 3.
We will discuss statistical significance and the relationship between sample size, effect size and power in more detail in 7 Sect. 4.5.
- 4.
A real cube is formed when three factors with two levels each are considered.
- 5.
Of course, the same procedure can be simulated faster on a computer.
- 6.
These data are only for instructional purposes and are not intended to be realistic.
- 7.
We will discuss an exception in the course of this section.
- 8.
It can also be shown that the retrospective power is about 50% if the p-value is equal to the significance level (Lenth 2007).
- 9.
Cummings calls this effect “Dance of the p-values” and demonstrates it on YouTube (e.g. 7 www.youtube.com/watch?v=5OL1RqHrZQ8)
- 10.
An example is a coin toss, in which “heads”, for instance, is defined as the case of success.
- 11.
It should be noted that the density function of the binomial distribution is only symmetrical when the probability of success or failure is 0.5. For all other values, the bars at the left and right end of the density function would have to be added up individually to obtain the p-value.
- 12.
To perform a multinomial test in R, for example, we need the EMT (Exact Multinomial Test) package from Uwe Menzel, which is available on every CRAN server. The execution is then done using the command multinomial.test().
- 13.
Because a single class (k = 1) does not provide any indication of a deviation between expectation and observation, the number of classes for the degrees of freedom is reduced by one.
- 14.
The example in this section will illustrate this using a numerical example.
- 15.
Unobservable variables are sometimes called latent variables.
- 16.
Of course, the intercept a also has an influence on the value of y. However, this influence is the same for all x values and only determines the overall level of the relationship. Therefore, a is not considered an effect.
- 17.
Later we will refine this overall random influence and explicitly model random effects.
- 18.
In the following, we will always indicate estimated values with a “hat”.
- 19.
Therefore we often refer to the “unexplained residual”.
- 20.
Two observations are always “useless” in this sense, since a regression line with only two observations will always go exactly through these two observed points, with the result that the residuals assume the value zero and, consequently, do not provide any information content with regard to a deviation.
- 21.
The wording “different from zero” is often omitted and then we only say, a parameter is “statistically significant”.
- 22.
Of course, it is not possible to draw any conclusions about the sign from the coefficient of determination, since both signs would be possible when calculating the square root.
- 23.
Odds represent the relationship between the probabilities of two opposing events. For example, p = 0.2 is the probability that a horse will win a horse race and 1–p = 0.8 is the probability that it will not win. Then the odds are 0.2/0.8 = 1/4 which are shown as odds 4:1 in continental European horse races. In case of a win you would get 4 Euro for every Euro you bet.
- 24.
An even clearer example would be a regression of “wage level” on “age”, with the first observations expected from an age of 16 years at the earliest. Without zero centering, a positive intercept of 300 euros, for example, would mean that newborns would receive an average wage of 300 euros.
References
Bliese, P., & Ployhart, E. E. (2002). Growth modeling using random coefficient models: Model building, testing, and illustrations. Organizational Research Methods, 5(4), 362–387.
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology & Evolution, 24(3), 127–135.
Bortz, J., & Lienert, G. A. (2008). Kurzgefasste Statistik für die klinische Forschung: Leitfaden für die verteilungsfreie Analyse kleiner Stichproben. 3. Auflage. Heidelberg: Springer.
Box, G. E. P., Hunter, J. S., & Hunter, W. G. (2005). Statistics for experimenters: design, innovation, and discovery (2nd ed.). New Jersey: John Wiley & Sons.
Brosig-Koch, J., Helbach, C., Ockenfels, A., & Weimann, J. (2011). Still different after all these years: Solidarity behavior in East and West Germany. Journal of Public Economics, 95(11–12), 1373–1376.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Lawrence Erlbaum Associates.
Conover, W. J. (1972). A Kolmogorov goodness-of-fit test for discontinuous distributions. Journal of the American Statistical Association, 67(339), 591–596.
Conover, W. J. (1973). On methods of handling ties in the Wilcoxon Signed-Rank test. Journal of the American Statistical Association, 68(344), 985–988.
Conover, W. J. (1999). Practical nonparametric statistics (3rd ed.). New York: John Wiley & Sons.
Cox, J. C., Friedman, D., & Gjerstad, S. (2007). A tractable model of reciprocity and fairness. Games and Economic Behavior, 59, 17–45.
Cumming, G. (2013). The new statistics: why and how. Psychological Science, 25(1), 7–29.
Davis, D. D., & Holt, C. A. (1993). Experimental economics. Princeton: Princeton University Press.
Dufwenberg, M., & Kirchsteiger, G. (2004). A theory of sequential reciprocity. Games and Economic Behavior, 47, 268–298.
Ellis, P.D. (2010). The essential guide to effect sizes. Cambridge et al.: Cambridge University Press.
Falk, A., & Fischbacher, U. (2006). A theory of reciprocity. Games and Economic Behavior, 54(2), 293–315.
Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions (3rd ed.). New York: John Wiley & Sons.
Griffiths, W. E., Hill, R. C., & Judge, G. G. (1993). Learning and practicing econometrics. New Jersey: John Wiley & Sons.
Gujarati, D., & Porter, D. (2008). Basic econometrics (5th ed.). Boston: McGraw-Hill.
Hilbe, J. M. (2009). Logistic regression models. London: Chapman & Hall/CRC.
Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24.
Hoffmann, S., Mihm, B., & Weimann, J. (2015). To commit or not to commit? An experimental investigation of pre-commitments in bargaining situations with asymmetric information. Journal of Public Economics, 121, 95–105.
Kanji, G. K. (2006). 100 statistical tests (3rd ed.). London: Sage Publications Ltd.
Kennedy, P. (2008). A guide to econometrics (6th ed.). Malden: Wiley-Blackwell.
Lee, Y., & Nelder, J. A. (2004). Conditional and marginal models: another view. Statistical Science, 19(2), 219–228.
Lenth, R. (2000). Two sample-size practices that I Don’t Recommend. Comments from panel discussion at the 2000 Joint Statistical Meetings in Indianapolis, http://www.stat.uiowa.edu/~rlenth/Power/2badHabits.pdf.
Lenth, R. (2001). Some practical guidelines for effective sample size determination. The American Statistician, 55(3), 187–193.
Lenth, R. (2007). Post Hoc power: Tables and commentary. Technical Report No. 378, University of Iowa, Department of Statistics and Actuarial Science, http://www.stat.uiowa.edu/files/stat/techrep/tr378.pdf.
Leonhart, R. (2008). Psychologische Methodenlehre/Statistik. München: UTB.
Levine, D. K. (1998). Modeling altruism and spitefulness in experiments. Review of Economic Dynamics, 1(3), 593–622.
Liang, K.-Y., & Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 78, 13–22.
Marascuilo, L. A., & McSweeney, M. (1977). Nonparametric and distribution-free methods for the social sciences. Monterey: Brooks/Cole.
Massey, F. J. (1951). The Kolmogorov-Smirnov test for goodness of fit. Journal of the American Statistical Association, 46, 68–78.
McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). Boca Raton: Chapman and Hall.
Morris, M. (2010). Design of experiments – an introduction based on linear models. London: Chapman and Hall.
Murphy, K. R., Myors, B., & Wolach, A. (2014). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (4th ed.). New York: Routledge.
Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A (General), 135(3), 370–384.
Nowak, M. A., & Siegmund, K. (2005). Evolution of indirect reciprocity. Nature, 437, 1291–1298.
Rabe-Hesketh, S., & Skrondal, A. (2010). Generalized linear mixed models. In P. Peterson, E. Baker, & B. McGaw (Eds.), International encyclopedia of education (pp. 171–177). Amsterdam: Elsevier.
Rabe-Hesketh, S., & Skrondal, A. (2004). Generalized latent variable modelling: Multilevel, longitudinal, and structural equation models. Boca Raton: Chapman & Hall/CRC.
Rabin, M. (1993). Incorporating fairness into game theory and economics. American Economic Review, 83, 1281–1302.
Sheskin, D. J. (2000). Parametric and nonparametric statistical procedures. Boca Raton: CRC Press.
Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences (2nd ed.). New York: McGraw-Hill.
Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society. Series B (Methodological), 13(2), 238–241.
Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis – modeling change and event occurrence. Oxford: Oxford University Press.
von Auer, L. (2016). Ökonometrie: Eine Einführung. 7. Auflage. Berlin, Heidelberg: Gabler Verlag.
Wu, C. F. J., & Hamada, M. S. (2009). Experiments: Planning, analysis, and optimization (2nd ed.). New York: John Wiley & Sons.
Zar, J. H. (1999). Biostatistical analysis (4th ed.). Upper Saddle River: Prentice Hall.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Weimann, J., Brosig-Koch, J. (2019). The Experiment from a Statistical Perspective. In: Methods in Experimental Economics. Springer Texts in Business and Economics. Springer, Cham. https://doi.org/10.1007/978-3-319-93363-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-93363-4_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-93362-7
Online ISBN: 978-3-319-93363-4
eBook Packages: Economics and FinanceEconomics and Finance (R0)