D-Optimal Design for Network A/B Testing
- 41 Downloads
A/B testing refers to the statistical procedure of experimental design and analysis to compare two treatments, A and B, applied to different testing subjects. It is widely used by technology companies such as Facebook, LinkedIn, and Netflix, to compare different algorithms, web-designs, and other online products and services. The subjects participating in these online A/B testing experiments are users who are connected in different scales of social networks. Two connected subjects are similar in terms of their social behaviors, education and financial background, and other demographic aspects. Hence, it is only natural to assume that their reactions to online products and services are related to their network adjacency. In this paper, we propose to use the conditional auto-regressive model to present the network structure and include the network effects in the estimation and inference of the treatment effect. A D-optimal design criterion is developed based on the proposed model. Mixed integer programming formulations are developed to obtain the D-optimal designs. The effectiveness of the proposed method is shown through numerical results with synthetic networks and real social networks.
KeywordsA/B testing Conditional auto-regressive model D-optimal design Mixed integer programming Social network
We sincerely thank the editor, the handling editor, and all the referees for their insightful comments which helped us improving the paper. This research was supported by US National Science Foundation Grant CMMI-1435902 and DMS-1916467.
Compliance with Ethical Standards
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
- 2.Atkinson AC, Woods DC (2015) Designs for generalized linear models, Chapter 13. In: Handbook of design and analysis of experiments. Chapman & Hall/CRC, Boca Raton, FL, pp 471–514Google Scholar
- 4.Basse GW, Airoldi EM (2015) Optimal model-assisted design of experiments for network correlated outcomes suggests new notions of network balance. ArXiv preprint arXiv:1507.00803
- 9.Bhat N, Farias VF, Moallemi CC, Sinha D (2017) Near optimal AB testing. Columbia Business School, New YorkGoogle Scholar
- 10.Bivand R, Bernat A, Carvalho M, Chun Y, Dormann C, Dray S, Halbersma R, Lewin-Koh N, Ma J, Millo G et al (2005) The spdep package. Comprehensive R Archive Network, Version 05–83Google Scholar
- 13.Draper N, Smith H (1966) Applied regression analysis. New York, Wiley, pp 108–116Google Scholar
- 14.Eckles D, Karrer B, Ugander J (2017) Design and analysis of experiments in networks: reducing bias from interference. J Causal Infer. https://doi.org/10.1515/jci-2015-0021
- 16.Gui H, Xu Y, Bhasin A, Han J (2015) Network a/b testing: from sampling to estimation. In: Proceedings of the 24th international conference on world wide web, international world wide web conferences steering committee, pp 399–409Google Scholar
- 20.Leskovec J, Mcauley JJ (2012) Learning to discover social circles in ego networks. In: Advances in neural information processing systems, pp 539–547Google Scholar
- 22.Nandy P, Basu K, Chatterjee S, Tu Y (2019) A/B testing in dense large-scale networks: design and inference. ArXiv preprint arXiv:1901.10505
- 23.Wolsey LA, Nemhauser GL (2014) Integer and combinatorial optimization. John Wiley & SonsGoogle Scholar
- 24.Nemhauser GL, Savelsbergh MWP, Sigismondi GS (1992) Constraint classification for mixed integer programming formulations. COAL Bull 20:8–12Google Scholar
- 25.Ogburn EL, Sofrygin O, Diaz I, van der Laan MJ (2017) Causal inference for social network data. ArXiv preprint arXiv:1705.08527
- 26.Pouget-Abadie J, Saveski M, Saint-Jacques G, Duan W, Xu Y, Ghosh S, Airoldi EM (2017) Testing for arbitrary interference on experimentation platforms. ArXiv preprint arXiv:1704.01190
- 29.Saveski M, Pouget-Abadie J, Saint-Jacques G, Duan W, Ghosh S, Xu Y, Airoldi EM (2017) Detecting network effects: randomizing over randomized experiments. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 1027–1035Google Scholar
- 30.Schmidt AM, Nobre WS (2014) Conditional autoregressive (CAR) model. Statistics reference online, Wiley StatsRef, pp 1–11Google Scholar
- 34.Xu Y, Chen N, Fernandez A, Sinno O, Bhasin A (2015) From infrastructure to culture: A/b testing challenges in large scale social networks. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 2227–2236Google Scholar