arXiv subjects

Elea McDonnell Feit

Publications and source records attributed to Elea McDonnell Feit.

3 recordsLinked to original sources

arXiv2023

Latent Stratification for Incrementality Experiments

Incrementality experiments compare customers exposed to a marketing action designed to increase sales to those randomly assigned to a control group. These experiments suffer from noisy responses which make precise estimation of the average treatment effect (ATE) and marketing ROI difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment -- customers who would buy regardless of ad exposure, those who would buy only if exposed to ads and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Analytical results and simulations show that the method decreases the sampling variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model used to estimate the strata-level effects is well-identified. Applying the procedure to 5 catalog experiments shows a reduction of 30-60% in the variance of the overall ATE. This leads to a substantial decrease in decision errors when the estimator is used to determine whether ads should be continued or discontinued.

stat.AP↗

arXiv2020

Context information increases revenue in ad auctions: Evidence from a policy change

Ad exchanges, i.e., platforms where real-time auctions for ad impressions take place, have developed sophisticated technology and data ecosystems to allow advertisers to target users, yet advertisers may not know which sites their ads appear on, i.e., the ad context. In practice, ad exchanges can require publishers to provide accurate ad placement information to ad buyers prior to submitting their bids, allowing them to adjust their bids for ads at specific domains, subdomains or URLs. However, ad exchanges have historically been reluctant to disclose placement information due to fears that buyers will start buying ads only on the most desirable sites leaving inventory on other sites unsold and lowering average revenue. This paper explores the empirical effect of ad placement disclosure using a unique data set describing a change in context information provided by a major private European ad exchange. Analyzing this as a quasi-experiment using diff-in-diff, we find that average revenue per impression rose when more context information was provided. This shows that ad context information is important to ad buyers and that providing more context information will not lead to deconflation. The exception to this are sites which had a low number of buyers prior to the policy change; consistent with theory, these sites with thin markets do not show a rise in prices. Our analysis adds evidence that ad exchanges with reputable publishers, particularly smaller volume, high quality sites, should provide ad buyers with site placement information, which can be done at almost no cost.

econ.GN↗

arXiv2019

Test & Roll: Profit-Maximizing A/B Tests

Marketers often use A/B testing as a tool to compare marketing treatments in a test stage and then deploy the better-performing treatment to the remainder of the consumer population. While these tests have traditionally been analyzed using hypothesis testing, we re-frame them as an explicit trade-off between the opportunity cost of the test (where some customers receive a sub-optimal treatment) and the potential losses associated with deploying a sub-optimal treatment to the remainder of the population. We derive a closed-form expression for the profit-maximizing test size and show that it is substantially smaller than typically recommended for a hypothesis test, particularly when the response is noisy or when the total population is small. The common practice of using small holdout groups can be rationalized by asymmetric priors. The proposed test design achieves nearly the same expected regret as the flexible, yet harder-to-implement multi-armed bandit under a wide range of conditions. We demonstrate the benefits of the method in three different marketing contexts -- website design, display advertising and catalog tests -- in which we estimate priors from past data. In all three cases, the optimal sample sizes are substantially smaller than for a traditional hypothesis test, resulting in higher profit.

stat.AP↗