Cambridge University Press Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Its imported from UK. Too costly

Too expensive.

This book is a masterpiece on Experimentation!!

The book is written in a very non-technical fashion and only includes writers' stories in their own companies practicing A/B testing. Many topics discussed throughout various chapters are redundant for the purpose of increasing the volume of book. Also, I believe that the book is over-priced compared to its poor print quality and small size!

A quick summary:A/B testing can be a powerful tool for IT companies to test out new ideas and provide data-driven evidence for innovation. However, without a correct mindset or appropriate design, A/B testing can go wrong. The corresponding results will not be trustworthy, and conclusions can be misleading and even damaging. The authors encourage readers to evaluate the trustworthiness of experiment results and share important practical lessons. This book will help a wide range of readers (e.g., executives, product managers, engineers, data scientists/analysts) to avoid the pitfalls and to make the most of A/B testing.More Detailed reading notesPart I is designed to be read by everyone, regardless of background. The basics about A/B testing (e.g., the concept of controlled experiments, the benefits) and the examples (Chapter 1 and Chapter 2) are relatively straightforward. The key message of Part I is that trustworthiness of (interpreting) experiment results is a critical issue. To increase the trust in experiments and experiment results, the authors laid out questions/suggestions from different levels:1. The company/executives level: three questions to answerDoes the company want to build a culture of making data-driven decisions?Is the company willing/ready to invest in the infrastructure and test to run A/B testing?Is the current way of assessing the value of ideas poor or not good enough?If yes, read the tenets (Page 11 to Page 14). More details in Chapters 4, 6, 7, and 8.2. The middle/design level (for senior manager, product managers)How to help build a robust and trustworthy experiment platform?How to design metrics and align them with missions/long-term goals (many goals contradict each other)?How to ensure that the results are trustworthy in general (internal validity and external validity)?Is A/B testing the only technique to provide evidence for data-driven decisions?Detailed answers are in Chapter 4 in Part I, Part II (Chapter 5 to Chapter 9) and Part III (Chapter 10 and Chapter 11).3. The operation level (for data analysts, engineers)The list of questions will be much longer and at a more detailed level. A few examples:How to process logs from multiple sources (e.g., logs from servers, logs from different client types)?How to choose a randomization unit (user level or session level)?How to conduct a power analysis?How to interpret the experiment results (e.g., p-value, confidence interval)?Details answers are in Part IV (Chapter 12 to Chapter 16) and Part V (Chapter 17 to Chapter 23).In sum, Part I introduces the basics about A/B testing and lays out the framework for the rest of the book.