The Problem with Data Driven Decisions

This week I look at the pitfalls of being data-driven. It seems simple, and logical, so why do we end up falling back on our gut feel? [blog.mindrocketnow.com]

Should I make my next investment decision based on objective data or taking a punt based on how I feel? Seems a straightforward answer, doesn’t it - who would want to be responsible for a significant financial decision that was made on a whim? (Kinda depends if it worked out…) But making a good data-driven decision is a lot harder than just intending to do so.

What are you measuring? This simple question unravels a chain of questions that turn out to each require careful thought. Let’s pretend we’re trying to figure out whether to spend money on a cool new app feature:

• What are the business (or strategic) outcomes you’re trying to influence?

• How does the app contribute to that outcome?

• How does the feature improve the ability of the app to contribute to that outcome?

• What metric can we put on that improvement?

• How can we measure those metrics? 

• Are those measurements allowed in our privacy guidelines?

• What would be the impact on those quantities if we didn’t build the app?

• Is there anything else we could do, at less cost, that would improve that metric, even in part?

• Is there anything else we could do, at same cost, that would improve an alternative, more important metric?

Most of the time, we don’t have a good answer to all of those questions, oftentimes because to do so thoroughly will take a lot of effort, more than the effort to ship the feature itself. So we end up making a best guess - aka taking a punt.

I’m really attracted to the concept of lean development, which codified learning by doing. The idea is to ship features as frequently as possible, measure impact upon metrics, and improve or discard depending on whether it’s a positive or negative impact. By keeping this feedback loop really short, we risk less wasted development effort. As a side-effect, we maintain focus on the metrics that matter to us, and maintain a cadence of shipping features.

As before, it depends upon the metric. If we optimise to a vanity metric (one that doesn’t align with a business objective) or a proxy metric (one that doesn’t align well with a business objective), then we might miss the business objective. We might optimise for a very fast playback start but miss the fact that consumers are much more worried about not finding programmes that they like.

After all that, you might be thinking that I advise avoiding making gut feel decisions. You’d be right - gut feel is basically your thought heuristics kicking in, reinforcing all your unconscious biases. Taking a moment to consider rather than react is a good rule. But that doesn’t mean you shouldn’t use your feelings. I’m also a strong believer in eating your own cooking. 

We should be building apps for people, and we need to understand people in detail if the app is going to make a difference to them. The person you get to see the most is yourself, so you should be able to analyse your own reactions to your app in the best detail.

Be data driven but guided by context. The best context is you.