Abstract
For the digital parts of businesses in Society 5.0, such as web site and mobile applications, manual testing is impractical and slow. Instead, implementation of ideas can now be evaluated with scientific rigor using online controlled experiments (A/B tests), which provide trustworthy reliable assessments of the impact of the implementations to key metrics of interest. This chapter shows how online controlled experiments can be run at large scale.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Box, G.E.P., Stuart Hunter, J., Hunter, W.G.: Statistics for Experimenters: Design, Innovation, and Discovery, 2nd edn. Wiley, Hoboken (2005)
Crook, T., Frasca, B., Kohavi, R., Longbotham, R.: Seven pitfalls to avoid when running controlled experiments on the web. KDD ‘09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105–1114. (2009)
Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., Vaz, G.: Pitfalls of long-term online controlled experiments. IEEE Int. Conf. Big Data. Washington, DC. 1367–1376 (2016). https://doi.org/10.1109/BigData.2016.7840744
Dmitriev, P., Gupta, S., Kim, D.W., Vaz, G.: A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2017), pp. 1427–1436. ACM, Halifax (2017). https://doi.org/10.1145/3097983.3098024
Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., Dmitriev, P.: Diagnosing sample ratio mismatch in online controlled experiments: a taxonomy and rules of thumb for practitioners. In: KDD ‘19: The 25th SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Anchorage (2019)
Gerber, A.S., Green, D.P.: Field Experiments: Design, Analysis, and Interpretation. W. W. Norton & Company, New York (2012). https://www.amazon.com/Field-Experiments-Design-Analysis-Interpretation/dp/0393979954
Gupta, S., Kohavi, R., Tang, D., Ya, X., et al.: Top challenges from the first practical online controlled experiments summit. 21(1). https://bit.ly/OCESummit1 (2019)
Hern, A.: Why Google has 200m reasons to put engineers over designers. The Guardian. Feb 5. https://www.theguardian.com/technology/2014/feb/05/why-google-engineers-designers (2014)
Hohnhold, H., O’Brien, D., Tang, D.: Focus on the long-term: it’s better for users and business. In: Proceedings 21st Conference on Knowledge Discovery and Data Mining (KDD 2015). ACM, Sydney (2015). http://dl.acm.org/citation.cfm?doid=2783258.2788583
Holson, L.M.: Putting a bolder face on google. NY Times. Feb 28. https://www.nytimes.com/2009/03/01/business/01marissa.html (2009)
Keidanren.: Society 5.0: co-creating the future. Keidanren, Japan Business Federation. http://www.keidanren.or.jp/en/policy/2018/095_booklet.pdf (2018)
Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., Xu, Y.: Trustworthy online controlled experiments: five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. http://bit.ly/expPuzzling (2012)
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., Pohlmann, N.: Online controlled experiments at large scale. KDD 2013: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. http://bit.ly/ExPScale (2013)
Kohavi, R., Deng, A., Longbotham, R., Xu, Y.: Seven rules of thumb for web site. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ‘14). http://bit.ly/expRulesOfThumb (2014)
Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Encyclopedia of Machine Learning and Data Mining, by Claude Sammut and Geoffrey I Webb. Springer, New York (2017)
Kohavi, R., Longbotham, R.: Unexpected results in online controlled experiments.” SIGKDD Explorations, Dec. http://bit.ly/expUnexpected (2010)
Kohavi, R., Thomke, S.: The surprising power of online experiments: getting the most out of A/B and other controlled tests. Har. Bus. Rev. (Sept-October). 95(5), 74–92 (2017)
Kohavi, R., Tang, D., Xu, Y.: Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press (2020). ISBN: 1108724264
Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Disc. 18, 140–181 (2009b). http://bit.ly/expSurvey.
Kohavi, R., Crook, T., Longbotham, R.: Online experimentation at Microsoft. In: Third Workshop on Data Mining Case Studies and Practice Prize. Association for Computing Machinery, Inc. (ACM), New York (2009a)
Siroker, D., Koomen, P.: A/B Testing: The Most Powerful Way to Turn Clicks into Customers. Wiley, Hoboken (2013)
Tang, D., Agarwal, A., O’Brien, D., Meyer, M.: Overlapping experiment infrastructure: more, better, faster experimentation. Proceedings 16th Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, Inc. (ACM), New York (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Kohavi, R. (2019). Online Controlled Experiments at Large Scale in Society 5.0. In: Fathi, M., Khakifirooz, M., Pardalos, P.M. (eds) Optimization in Large Scale Problems. Springer Optimization and Its Applications, vol 152. Springer, Cham. https://doi.org/10.1007/978-3-030-28565-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-28565-4_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28564-7
Online ISBN: 978-3-030-28565-4
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)