Abstract
This section describes how power analysis on published papers can be done using a suite of simple R scripts, so that better-designed experiments can be conducted in the future. Here, “better” means “ensuring appropriate statistical power”. First, an overview of the five R scripts is given (Sect. 7.2), followed by a description of each script (Sects. 7.3, 7.4, 7.5, 7.6, and 7.7). The five scripts, which are for paired t-test, two-sample t-test, one-way ANOVA, two-way ANOVA without replication, and two-way ANOVA with replication, respectively, were adapted from the R scripts of Toyoda (Introduction to statistical power analysis: a tutorial with R (in Japanese). Tokyo Tosyo, 2009): his original scripts, which contain Japanese character codes, are available from his book’s website (http://www.tokyo-tosho.co.jp/download/DL02065.zip); Toyoda’s scripts (and therefore mine as well) rely on R libraries called stats and pwr. (The present author is solely responsible for any problems caused by modifying the original scripts of Toyoda.) Finally, it provides summary while touching upon a survey I conducted using these R scripts, with a decade’s worth of IR papers from ACM SIGIR (http://sigir.org/) and TOIS (https://tois.acm.org/) (Sakai Statistical significance, power, and sample sizes: a systematic review of SIGIR and TOIS. In: Proceedings of ACM SIGIR 2016, pp 5–14, 2016), where it was demonstrated that there are highly overpowered and highly underpowered experiments in the results reported in the IR literature. Highly overpowered experiments use a lot more resources than necessary, while highly underpowered experiments are highly likely to miss important differences that exist due to the use of small samples. We can probably do better by learning from previous studies and/or from pilot studies.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Note that you can look up the specification of any standard R function using ? on the R command line, e.g. ? ‘power.t.test’.
- 2.
If the group sizes are unequal, the average group size over the m groups can be used [4].
- 3.
This is in fact the example we discussed in Sect. 7.3.
- 4.
For general considerations required for designing user studies, see Kelly [2].
References
J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd edn. (Psychology Press, New York, 1988)
D. Kelly, Methods for evaluating interactive information retrieval systems with users. Found. Trends Inf. Retr. 3(1–2), 1–224 (2009)
T. Sakai, Statistical significance, power, and sample sizes: a systematic review of SIGIR and TOIS, in Proceedings of ACM SIGIR, Pisa, 2016, pp. 5–14
H. Toyoda, Introduction to Statistical Power Analysis: A Tutorial with R (in Japanese) (Tokyo Tosyo, Chiyoda, 2009)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Sakai, T. (2018). Power Analysis Using R. In: Laboratory Experiments in Information Retrieval. The Information Retrieval Series, vol 40. Springer, Singapore. https://doi.org/10.1007/978-981-13-1199-4_7
Download citation
DOI: https://doi.org/10.1007/978-981-13-1199-4_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1198-7
Online ISBN: 978-981-13-1199-4
eBook Packages: Computer ScienceComputer Science (R0)