Auxiliary Decision-making for Controlled Experiments based on Mid-term Treatment Effect Prediction: Applications in Ant Financial’s Offline-payment Business

Gang Li, Huizhi Xie

2020

Abstract

Controlled experiments are commonly used in technology companies for product development, algorithm improvement, marketing strategy evaluation, etc. These experiments are usually run for a short period of time to enable fast business/product iteration. Due to the relatively short lifecycle of these experiments, key business metrics that span a longer window cannot be calculated and compared among different variations of these experiments. This is essentially a treatment effect prediction issue. Research in this paper focuses on experiments in the offline-payment business at Ant Financial. Experiments in this area are usually run for one or two weeks, sometimes even shorter, yet the accumulating window of key business metrics such as payment days, payment counts is one month. In this paper, we apply the classic BG/NBD model(Fader et al., 2005) in marketing to predict users payment behavior based on data collected from the relatively short experimentation periods. The predictions are then used to evaluate the impact on the key business metrics. We compare this method with supervised learning methods and direct modelling of treatment effect as a time series. We show the advantage of the proposed method using data collected from plenty of controlled experiments in Ant Financial. The proposed technique has been integrated into Ant Financial experimentation reporting platform, where metrics based on the predictions are one of the auxiliary evaluation criteria in offline-payment experiments.

Paper Citation

in Harvard Style

Li G. and Xie H. (2020). Auxiliary Decision-making for Controlled Experiments based on Mid-term Treatment Effect Prediction: Applications in Ant Financial’s Offline-payment Business.In Proceedings of the 9th International Conference on Data Science, Technology and Applications - Volume 1: DATA, ISBN 978-989-758-440-4, pages 19-30. DOI: 10.5220/0009770500190030

in Bibtex Style

@conference{data20,
author={Gang Li and Huizhi Xie},
title={Auxiliary Decision-making for Controlled Experiments based on Mid-term Treatment Effect Prediction: Applications in Ant Financial’s Offline-payment Business},
booktitle={Proceedings of the 9th International Conference on Data Science, Technology and Applications - Volume 1: DATA,},
year={2020},
pages={19-30},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009770500190030},
isbn={978-989-758-440-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 9th International Conference on Data Science, Technology and Applications - Volume 1: DATA,
TI - Auxiliary Decision-making for Controlled Experiments based on Mid-term Treatment Effect Prediction: Applications in Ant Financial’s Offline-payment Business
SN - 978-989-758-440-4
AU - Li G.
AU - Xie H.
PY - 2020
SP - 19
EP - 30
DO - 10.5220/0009770500190030