The following is adapted from Build What Matters.
Octopus Interactive is a rideshare media company based in Washington, DC. Co-Founder and COO Brad Sayler explains the history of the company and an early product feedback issue they encountered their first few years:
“Most of our team members started our product careers with a lifestyle app called Spotluck that helped consumers decide where to eat and save money at restaurants. This app was fairly involved, so relied on consumers to adapt and learn it over time. The feedback loop was long – anywhere from thirty to sixty days depending on the complexity of the new feature we’d introduced. To further confound our analysis, the product wasn’t intended to be used every-day or even every-week, so when usage varied we had difficulty attributing it to product changes vs. seasonality vs. folks simply forgetting to use it. Thus, we had no reliable measurement to gauge user adoption/satisfaction.”
Prodify began working with Brad and his product team during the pivot from Spotluck to Octopus, and Brad knew they would finally have a short feedback loop they could use to inform product iterations. As he put it:
“For our next product, we transitioned to a simpler, briefer experience – a micro-experience, wherein we installed entertainment/advertising screens in the back of rideshare vehicles. No app download required, no need to be front-of-mind, simply look up and tap the screen if you’d like to interact. We only have a few minutes to capture someone’s attention, but this micro-experience provides us a rapid, strong feedback loop.”
But what was the right metric to use to know if this tablet was working? Octopus had an interesting choice to make, as their product served many stakeholders who all sought different outcomes:
- Brands and their agencies who wanted to build awareness, drive traffic, or collect leads
- Drivers who wanted to provide a better rider experience to increase tips and ratings
- Riders who wanted a change of pace from staring out the window or at their phone
In the end, Octopus chose a product metric that has stood the test of time as a stable indicator of value for all stakeholders. In Brad’s words,
“By measuring ‘content interactions per driver hour,’ we can quickly see if a new feature boosts user engagement. This quantitative measurement gives us nearly real-time accurate feedback – results on day one of a new feature are similar to results on day thirty or sixty.”
The beauty of the “content interactions per driver hour” metric is that it was a simple, durable metric that encompassed the outcomes sought by every Octopus stakeholder:
- Brands want to know how many riders are engaging with their content and how often
- Drivers want to offer an engaging experience riders (normalizing by “driver hour” allows Octopus to track the rate of engagement over time, even if driver hours fluctuated)
- Riders want to engage with the tablet if it looks “interesting” (note that the definition of an interaction could change over time - in response to COVID-19, the Octopus team tested Trivia Party, a way to play their most-popular trivia game from your own phone as a way to engage with the tablet without touching the screen)
With this metric in mind, Octopus built their product team and operations around optimizing it. They created operational tools that let product designers and product managers create new games and ads for riders without involving engineering. They split every city into two cohorts of drivers so they could run A/B tests over the same time period and know the city wouldn’t influence experiment results. They used “content interactions per driver hour” to monitor if and how engagement changed when they integrated with ad tech markets to fill their ad capacity. As Brad concluded,
“Our rapid feedback loop, combined with a reliable measurement technique, allows us to continually iterate and optimize our product with phased roll-outs (and roll-backs when needed) and A/B testing.”
