Wald’s example was running two manufacturing processes and then shutting down the one that seems to have a lower success rate. The idea is so brilliant, it is easier to demonstrate than to describe (so we suggest skipping ahead and checking out our worked example here). It is a chart you can actually print, laminate, and post on the machine shop floor. This is completely procedural, but brilliant. When you cross one of the decision lines you are done (“Reject process 1” means go with 2, and “Accept process 1” means go with 1). When you see a (0,1) or (1,0) pair move on the inspection chart one unit to the right if the observation was a (0,1) also move one unit up (a process 1 “win”). Start at the origin of the inspection chart.
PARTIAL AND SEQUENTIAL TESTING IN R UPDATE
Only look at pairs (0,1) or (1,0) (we discard all (0,0) and (1,1) pairs, they are not allowed to update the chart). We get back pair measurements (c1,c2) where c1=1 if process 1 paid off (0 otherwise) and c2=1 if process 2 paid off (zero otherwise). The chart is used as follows: we send traffic to processes 1 and 2 in matched pairs. With these parameters Wald designs an inspection plan that is a single chart (shown below).įigure 14, Section 6.4.2, page 111, Abraham Wald, Sequential Analysis, Dover 2004 (reprinting a 1947 edition). Wald treats A/B testing as a “stopping problem.” Wald asks the business partner for four parameters: u0,u1 (the desired bounds on relative error allowed in the estimate) and alpha,beta (the power and significance goals). Wald’s graphical sequential inspection procedureĪ particularly compelling (and unusual) type of test plan is the graphical sequential inspection procedure designed by Abraham Wald. In this “statistics as it should be” article we will discuss Wald’s sequential analysis. If you have ever heard of a test plan such as “first process to get more than 30 wins ahead of the other is the one we choose” you have seen methods derived from Wald’s sequential analysis technique. That clever idea is called “sequential analysis” and was introduced by Abraham Wald (somebody we have written about before). Explicitly solving such dynamic programs gets long and tedious, so you are well served by finding and introducing clever invariants to track (something better than just raw win-rates). Our most recent article was a dynamic programming solution to the A/B test problem.
PARTIAL AND SEQUENTIAL TESTING IN R SERIES
We here at Win-Vector LLC been working through an ad-hoc series about A/B testing combining elements of both operations research and statistical points of view.