Investigating the Analysis of Data from Adaptive Experiments
Abstract
People increasingly interact with software to achieve their goals in the world websites, apps, emails, messages that help them learn, make decisions, communicate. Randomized experiments are increasingly used to test out different versions of experiences, to discover how to enhance how software helps people, as well as to discover principles about how people behave and how to design technology. There is the opportunity for machine learning algorithms to be used for adaptive experimentation, where the assignment of alternative arms (conditions, versions) in an experiment is changed over time as data is collected. There is extensive theoretical work onmulti-armed bandit algorithms and other techniques that are relevant to rapidly using data to give future users or participants in an experiment better arms (Lattimore & Szepesvri, 2020), and there are increasing applications of algorithms in industry (e.g. the widely used Thompson Sampling, Chapelle & Li, 2011; Scott, 2015). However, these have raised a number of questions about doing statistical analysis of data collected by bandit algorithms (and related approaches). The proposed work will explore how to more reliablyanalyze data collected by algorithms for adaptive experimentation. Our approach involves: (1) characterizing these biases in specific, real-world adaptive experiments, (2) modifying algorithms for adaptive experimentation to better address the needs and goals of statistical tests, and (3) modifying statistical tests to better incorporate the properties of algorithms for adaptive experimentation.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Aug 05, 2021
- Source ID
- N000142112576
Entities
People
- Joseph Williams
Organizations
- Office of Naval Research
- United States Navy
- University of Toronto