On Bayes Factors for Success Rate A/B Testing

Maciej Skorski

2019

Abstract

This paper discusses Bayes factors, an alternative to classical frequentionist hypothesis testing, within the standard A/B proportion testing setup - observing outcomes of independent trails (which finds applications in industrial conversion testing). It is shown that the Bayes factor is controlled by the Jensen-Shannon divergence of success ratios in two tested groups, and the latter one is bounded (under mild conditions) by Welch’s t-statistic. The result implies an optimal bound on the necessary sample size for Bayesian testing, and demonstrates the relation to its frequentionist counterpart (effectively bridging Bayes factors and p-values).

Paper Citation

in Harvard Style

Skorski M. (2019). On Bayes Factors for Success Rate A/B Testing.In Proceedings of the 8th International Conference on Data Science, Technology and Applications - Volume 1: DATA, ISBN 978-989-758-377-3, pages 332-339. DOI: 10.5220/0007954503320339

in Bibtex Style

@conference{data19,
author={Maciej Skorski},
title={On Bayes Factors for Success Rate A/B Testing},
booktitle={Proceedings of the 8th International Conference on Data Science, Technology and Applications - Volume 1: DATA,},
year={2019},
pages={332-339},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007954503320339},
isbn={978-989-758-377-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 8th International Conference on Data Science, Technology and Applications - Volume 1: DATA,
TI - On Bayes Factors for Success Rate A/B Testing
SN - 978-989-758-377-3
AU - Skorski M.
PY - 2019
SP - 332
EP - 339
DO - 10.5220/0007954503320339