The following is adapted from Build What Matters.
One of the most fundamental experiences in an e-commerce product is called “finding,” showing buyers the items they’re looking for. At eBay, I was product manager on the finding team responsible for the item details page, which showed what sellers listed. Counterparts on my team included product managers for searching, browsing, the bidding flow, and so on. Our team collaborated well, worked hard, and learned a great deal from one another. We also made our share of mistakes.
As we were all data-driven, we wanted to quantify the quality of our search experience so we could improve it over time, so we developed an internal metric: number of items viewed per search conducted. The idea was that if a customer ran a search but failed to click on any of the results, we must have missed the mark. However, if a customer clicked on a whole bunch of the items shown, then we must have nailed it. On the surface, this made sense, but we later learned the hard way that it was an awful metric to manage against. Here’s why.
At the time, shipping costs weren’t shown on the search results page. Buyers could see the price of an item immediately, but they needed to click on the item to see the additional shipping cost, which was only available on the item page. We knew this was an annoying buyer experience, because buyers would see what appeared to be a great deal, excitedly click on it, then feel disappointed when they saw an exorbitant shipping charge. So they would click the browser back button, return to the search results page, and select the next item, and the next item, and so on. I watched a usability study in which a customer asked for a pad of paper to write down the shipping costs in order to manually add them to the item prices and find the best deal. Ouch!
Making matters worse, sellers who wanted more clicks on their items competed to have the lowest prices displayed on search results. Since they knew shipping costs weren’t displayed, they realized that the best way to sell their products was to build all of their profit margin into shipping, so they jacked up shipping costs to unreasonably high levels. It wasn’t unusual to find brand new DVDs selling for $0.01 with $25 ground shipping.
Why didn’t we immediately prioritize displaying shipping costs on search results? The answer was surprisingly simple: we didn’t want to. We had established the “items viewed per search” metric with our buyers’ best interests in mind, and we became devoted to it. We knew that showing high shipping costs would reduce total item clicks, and we didn’t want to unravel the progress that had been made through hard work.
Despite the absurdity of this thinking, the logic was sound. The problem lay in the metric we were trying to optimize. It didn’t directly measure what our customers actually wanted. If we had asked them, no one would have said, “I want to click on as many items as I can every time I run a search.” They would have said, “I want it to be fast, easy, and fun to find what I’m looking for.” That’s what we should have been measuring from the beginning.
Had we framed success from our customers’ perspective, rather than using our internal lens, we would have quickly come to a different conclusion, fixed what was broken sooner, and delivered a more delightful experience that drove buyer loyalty. Instead, we sent them packing to this other site you may have heard of called Amazon.com.
Are you trying to create a strong product vision? Click here to learn more about our 3 most popular free sources and sign up to have them delivered to your inbox.
