+353-1-416-8900REST OF WORLD
+44-20-3973-8888REST OF WORLD
1-917-300-0470EAST COAST U.S
1-800-526-8630U.S. (TOLL FREE)

PRINTER FRIENDLY

Business Experiments with R. Edition No. 1

  • Book
  • 384 Pages
  • April 2021
  • John Wiley and Sons Ltd
  • ID: 5186121
BUSINESS EXPERIMENTS with R

A unique text that simplifies experimental business design and is dedicated to the R language

Business Experiments with R offers a guide to, and explores the fundamentals of experimental business designs. The book fills a gap in the literature to provide a text on the topic of business statistics that addresses issues such as small samples, lack of normality, and data confounding. The author - a noted expert on the topic - puts the focus on the A/B tests (and their variants) that are widely used in industry, but not typically covered in business statistics textbooks.

The text contains the tools needed to design and analyze two-treatment experiments (i.e., A/B tests) to answer business questions. The author highlights the strategic and technical issues involved in designing experiments that will truly affect organizations. The book then builds on the foundation in Part I and expands the multivariable testing. Since today’s companies are using experiments to solve a broad range of problems, Business Experiments with R is an essential resource for any business student. This important text: - Presents the key ideas that business students need to know about experiments - Offers a series of examples, focusing on a specific business question - Helps develop the ability to frame ill-defined problems and determine what data and analysis would provide information about that problem

Written for students of general business, marketing, and business analytics, Business Experiments with R is an important text that helps to answer business questions by highlighting the strategic and technical issues involved in designing experiments that will truly affect organizations.

Preface xiii

Acknowledgments xvii

Bruce McCullough xix

About the Companion Website xxi

1 Why Experiment? 1

1.1 Case: Life Expectancy and Newspapers 2

Exercises 6

1.2 Case: Credit Card Defaults 7

1.2.1 Lurking Variables 9

1.2.2 Sample Selection Bias 11

Exercises 13

1.3 Case: Salk Polio Vaccine Trials 14

Exercises 17

1.4 What Is a Business Experiment? 17

1.4.1 Four Steps of an Experiment 21

1.4.2 Big Three of Causality 22

1.4.3 Most Experiments Fail 23

Exercises 24

1.5 Improving Website Designs 24

Exercises 30

1.6 A Brief History of Experiments 31

1.7 Chapter Exercises 34

1.8 Learning More 34

1.9 Statistics Refresher 38

2 Analyzing A/B Tests: Basics 43

2.1 Case: Improving Response to Sales Calls (Two-Sample Test of Means) 44

2.1.1 Initial Analysis and Visualization 45

2.1.2 Confidence Interval for Difference Between Means 47

2.1.3 Reporting Results 52

2.1.4 Hypothesis Test for Comparing Means 52

2.1.5 Power and Sample Size for Tests of Difference of Means 56

2.1.6 Considering Costs 60

Exercises 62

2.2 Case: Email Response Test (Two-Sample Test of Proportions) 64

2.2.1 Confidence Interval and Hypothesis Test for Comparing Two Proportions 66

2.2.2 Better Confidence Intervals for Comparing Two Proportions 66

2.2.3 Power and Sample Size for Tests of Difference of Two Proportions 68

Exercises 70

2.3 Case: Comparing Landing Pages (Two-Sample Test of Means, Again) 71

Exercises 74

2.4 Case: Display Ad Clickthrough Rate 75

2.4.1 Beta-Binomial Model 75

2.4.2 Comparing Two Proportions Using the Beta-Binomial Model 78

Exercises 80

2.5 Case: Hotel Ad Test 81

2.5.1 Tips on Presenting Experimental Findings 83

Exercises 84

2.6 Chapter Exercises 84

2.7 Learning More 86

3 Designing A/B Tests with Large Samples 91

3.1 The Average Treatment Effect 92

Exercises 93

3.2 Internal and External Validity 93

3.2.1 Threats to Internal Validity 93

3.2.2 Threats to External Validity 95

Exercises 96

3.3 Designing Conclusive Experiments 96

Exercises 102

3.4 The Lady Tasting Tea 103

Exercises 103

3.5 Testing a New Checkout Button 104

Exercises 104

3.6 Chapter Exercises 104

3.7 Learning More 104

4 Analyzing A/B Tests: Advanced Techniques 107

4.1 Case: Audio/Video Test Reprise (One-Sided Tests) 108

4.1.1 One-Sided Confidence Intervals 109

4.1.2 One-Sided Power 112

Exercises 113

4.2 Case: Typing Test (Paired t-Test) 114

4.2.1 Matched Pairs 115

Exercises 119

4.3 A/B/n Tests 121

Exercises 126

4.4 Minimum Detectable Effect 126

Exercises 128

4.5 Subgroup Analysis 129

4.5.1 Deficiencies of Subgroup Analysis 132

4.5.2 Subgroup Analysis of Bank Data 133

Exercises 135

4.6 Simpson’s Paradox 136

4.6.1 Sex Discrimination at UC Berkeley 137

4.6.2 Do You Want Kidney Stone Treatment A or Treatment B? 138

4.6.3 When the Subgroup Is Misleading 140

Exercises 143

4.7 Test and Roll 143

Exercises 145

4.8 Chapter Exercises 146

4.9 Learning More 146

4.10 Appendix on One-Sided CIs, Tests, and Sample Sizes 151

5 Designing Tests with Small Samples 159

5.1 Case: Call Center Scripts (ANOVA) 160

5.1.1 Blocking 161

Exercises 165

5.2 Case: Facebook Geo-Testing (Latin Square Design) 166

5.2.1 More on Latin Square Designs 169

5.2.2 Latin Squares and Degrees of Freedom 172

Exercises 175

5.3 Dealing with Covariate Imbalance 177

5.3.1 Matching 179

5.3.2 Rerandomization 180

5.3.3 Propensity Score 181

5.3.4 Optimal Matching 184

5.3.5 Sophisticated Matching: Selling Slushies 184

Exercises 185

5.4 Chapter Exercises 187

5.5 Learning More 188

6 Analyzing Designs via Regression 193

6.1 Experiments and Linear Regression 193

Exercises 198

6.2 Dummies, Effect Coding, and Orthogonality 198

Exercises 203

6.3 Case: Loan Experiment Revisited (Interactions) 203

6.3.1 Interactions 203

6.3.2 Loan Experiment 208

Exercises 215

6.4 Case: Direct Mail (Three-Way Interactions) 215

Exercises 224

6.5 Pretreatment Covariates in Regression 224

Exercises 225

6.6 Chapter Exercises 226

6.7 Learning More 228

6.8 Appendix: The Covariance Matrix of the Regression Coefficients 233

7 Two-Level Full Factorial Experiments 237

7.1 Case: The Postcard Example 238

Exercises 247

7.2 Case: Email Campaign 247

Exercises 250

7.3 The Determinant of a Matrix 252

Exercises 257

7.4 Aliasing 258

Exercises 264

7.5 Blocking (Again) 265

Exercises 269

7.6 Mee’s Blunders 269

7.7 Chapter Exercises 270

7.8 Learning More 271

7.9 Appendix on aliasMatrix and colorMap 273

8 Two-Level Screening Designs 279

8.1 Preliminaries 280

Exercises 286

8.2 Case: Puncture Resistance (Small Screening Experiment) 287

Exercises 288

8.3 Case: College Giving (Big Screening Experiment) 289

Exercises 292

8.4 How to Set Up a Screening Experiment 294

Exercises 295

8.5 Creating a Screening Design 295

Exercises 298

8.6 Chapter Exercises 299

8.7 Learning More 300

9 Custom Design of Experiments 305

9.1 Case: Selling Used Cars at Auction I (Small Custom Screening) 306

9.1.1 Create the Design 307

9.1.2 Evaluate the Design 312

9.1.3 Use the Design 316

Exercises 319

9.2 Case: Selling Used Cars at Auction II (Custom Experiment) 319

Exercises 322

9.3 Custom Experiment with Blocking 322

Exercises 324

9.4 Custom Screening Experiments 326

Exercises 331

9.5 More Than Two Levels 332

Exercises 337

9.6 Chapter Exercises 338

9.7 Learning More 338

10 Epilogue 341

10.1 The Sequential Nature of Experimentation 342

10.2 Approaches to Sequential Experimentation 345

References 347

Index 357