Ronny Kohavi (Microsoft) started out by telling a famous true story about Greg Linden’s experience moving a recommender to the shopping cart at Amazon. A Senior VP of Marketing vetoed Greg’s proposal fearing that it would distract customers from checking out and paying for the items already in their shopping basket reducing conversion. This is where the “HiPPO” in the title of Ronny’s presentation comes from. It stands for the “Highest Paid Person’s Opinion” and sometimes for the person (e.g., the VP) holding the opinion. The Amazon story had a happy ending because Jeff Bezos had established a corporate culture that allowed for experiments to be run so Greg was able to run an experiment to test the hypothesis of the HiPPO. It turned out that conversions did indeed drop but the increased revenue due to customers purchasing recommended items was substantially greater than the loss.
The online controlled experiments advocated by Ronny including AB tests are like the trials used to test drugs and get at the causes of observed effects since all the experimental subjects are exposed to the same non-causal factors. Ronny ran the audience through a series of examples showing how difficult it is to make correct decisions about a series of alternative web page designs. Since people are bad at evaluating proposals, especially in evaluating more novel innovations, it is important to test a lot of ideas, “fail fast” and try again quickly instead of doing elaborate planning and preparation in advance.
In Ronny’s years of experience, he has observed that people and organizations go through stages: they go from hubris to getting insights through measurement followed by the “Semmelweis Reflex” and then fundamental understanding. Initially, people are sure they know it all but then they realize it helps to do experiments and take measurements to get data. The Semmelweis reflex is the reflex-like rejection of new knowledge because it contradicts entrenched beliefs. It is named after Ignaz Semmelweis who proposed that doctors clean their hands to reduce the spread of infections but who was rejected in spite of the fact that he had data to support his claim.
Ronny’s “take home” points were:
- data trumps intuition – it’s hard to assess the value of ideas so do experiments;
- get your organization to agree on what to optimize and use data to drive decisions.
A video recording of Ronny’s presentation is available at dyyno.com. More technical information including information on how to conduct experiments is available at http://www.exp-platform.com.