Trustworthy Online Controlled Experiments

Love how detailed this book is about all the subtleties and pitfalls in experimentaion.

Online Controlled Experiments (A/B Testing) is a game-changer when it comes to decision-making, creating compelling customer experiences, and ensure maximum return of investment (ROI).A/B testing gives small and large digital companies alike the power to give their users a voice and let them, by their actions, steer the direction of all digital development.As a concept, Online Controlled Experiments (A/B Testing) is simple. Two or more variants of a webpage or app are shown to users at random to determine which one performs better. However, the execution is often filled with frustration, and companies make serious mistakes.For experienced A/B test practitioners, this is not surprising. A/B testing is reliant upon proper statistical practices and there are many other potential pitfalls besides pure statistics one encounter.Also, building a data-driven culture around digital experiment is far from easy. It requires that you have management on board and that they are okay both by being proven wrong by data and to delegate power to run experiments to the people working for them.I have personally found it next to impossible to find one affordable single source of truth on how to properly conduct A/B tests, learn to identify (and mitigate) the risks one often encounter when doing so while also providing the information on how to communicate and persuade management to give A/B testing the conditions to thrive.This book change that.Within the small community of people dedicated to A/B testing, the authors are among the most well-known and cited authorities.The book is divided into different sections. At the beginning of the book, it is explained what audience each section is suitable for.The result is a book valuable for anyone interested in the field of A/B Testing. It is easy to read, but it is also "meaty", with a lot of good advice for both new and experienced A/B testing practitioners.For management and business readers without any prior knowledge, a lot of high-level examples are provided. Since the authors, all have wast experience from digital giants such as Google, LinkedIn, and Microsoft, these examples give a rare glimpse into how digital experiments are conducted in real life within these companies and also the amazing results achieved by doing so.Something I especially like is that this book is research-based with a clear reference to academic papers (which one can dig deeper into if there is an interest in doing so).To sum it up, this book is the best one I've read so far on the topic of A/B testing and digital experimentation and I highly recommend it to anyone interested in this field.Note: during the authors' writing process, I have had the opportunity to give them feedback. For this, I have been mentioned in the acknowledgment section of the book and I have also been given a physical copy free of charge. Still, as one can tell by my review, I choose to purchase a digital copy just to ensure I can constantly access the book through my Kindle and computer when I don't have the physical copy at hand. This by itself should give you an indication of how valuable I consider the information in it to be.

A quick summary:A/B testing can be a powerful tool for IT companies to test out new ideas and provide data-driven evidence for innovation. However, without a correct mindset or appropriate design, A/B testing can go wrong. The corresponding results will not be trustworthy, and conclusions can be misleading and even damaging. The authors encourage readers to evaluate the trustworthiness of experiment results and share important practical lessons. This book will help a wide range of readers (e.g., executives, product managers, engineers, data scientists/analysts) to avoid the pitfalls and to make the most of A/B testing.More Detailed reading notesPart I is designed to be read by everyone, regardless of background. The basics about A/B testing (e.g., the concept of controlled experiments, the benefits) and the examples (Chapter 1 and Chapter 2) are relatively straightforward. The key message of Part I is that trustworthiness of (interpreting) experiment results is a critical issue. To increase the trust in experiments and experiment results, the authors laid out questions/suggestions from different levels:1. The company/executives level: three questions to answerDoes the company want to build a culture of making data-driven decisions?Is the company willing/ready to invest in the infrastructure and test to run A/B testing?Is the current way of assessing the value of ideas poor or not good enough?If yes, read the tenets (Page 11 to Page 14). More details in Chapters 4, 6, 7, and 8.2. The middle/design level (for senior manager, product managers)How to help build a robust and trustworthy experiment platform?How to design metrics and align them with missions/long-term goals (many goals contradict each other)?How to ensure that the results are trustworthy in general (internal validity and external validity)?Is A/B testing the only technique to provide evidence for data-driven decisions?Detailed answers are in Chapter 4 in Part I, Part II (Chapter 5 to Chapter 9) and Part III (Chapter 10 and Chapter 11).3. The operation level (for data analysts, engineers)The list of questions will be much longer and at a more detailed level. A few examples:How to process logs from multiple sources (e.g., logs from servers, logs from different client types)?How to choose a randomization unit (user level or session level)?How to conduct a power analysis?How to interpret the experiment results (e.g., p-value, confidence interval)?Details answers are in Part IV (Chapter 12 to Chapter 16) and Part V (Chapter 17 to Chapter 23).In sum, Part I introduces the basics about A/B testing and lays out the framework for the rest of the book.

I like the full spectrum of topics about controlled experiments covered in this book, nicely organized in 23 chapters. Experimentation is a core and fundamental piece of Data Science work in almost all companies that is equipped with DL and DS capabilities. Technical challenges & solutions, best practices, and related stats, however, exist in mostly scattered style, from my personal experiences. This book provides a comprehensive overview of the topic.I wish that there could be case study walk through about the experiment design and experiment data analysis.