When Should You Be Data-Driven?
Being data-driven is a default mode in product management. Have hypotheses to improve your
More and more, I see folks challenging the definition of being data-driven. Because justifying every product decision with numbers doesn’t automatically lead to success. To make it work, we need to adapt the data-driven process to product and market realities.
Underlying principles
Data-based product decisions are built on core principles that PMs must consider.
The Cost of Data
It’s often too expensive to gather all the information for a product decision. You have to collect it, clean, process, validate it — and only then you may get actionable insights. Yet your timelines demand decisions now. That’s why it is more practical to act based on the relatively small dataset you already have. The key phrase here is “relatively small”. How much information do you actually need to make a decision? Jeff Bezos, in his 2016 Letter to Shareholders, mentioned a 70% threshold:
Most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you’re probably being slow.
Timing of Product Decision
Beyond cost, timing is another critical factor. We have entered an ultra-fast era of product management. The near-zero cost and rapid speed of launching a product — thanks to LLMs and coding assistants — have changed the game. The assumptions and data you used when making a decision may already be outdated by the time you’re executing on it.
To avoid such cases, you need to identify which pieces of information are likely to go obsolete quickly and could make your previous decision suboptimal.
Data Isn’t Enough
Even having all the data doesn’t guarantee high-quality decisions. There’s a missing piece: the decision-making process itself. You should think about how the decisions will be made, what information you need, who should be included from your team and company, and which product, business and guardrail metrics will help you to monitor and correct the decision. Prepare and agree with stakeholders on a decision-making process early since delaying this may lead you to a waste of time.
So when and how to be data-driven?
Estimating vs Shipping
Sometimes, the cost of launching the actual product is lower than conducting research, gathering the data and estimating its impact beforehand. As mentioned earlier, LLM and AI agents have made it incredibly cheap to launch the first version of a product. The faster you ship, the faster you learn, validate and improve. Why spend time on numerous meetings where different product teams are trying to persuade each other using their projections in Excel or PowerPoint. Ship it and learn real numbers from customers.
Type of Decision
Not all product decisions are equal. You can categorise them by opportunity cost, cost/probability of failure, or whether they’re reversible. In the last category, Jeff Bezos distinguished between Type 1 and Type 2 decisions. Type 1 decisions are irreversible — “you can’t get back to where you were before.” And Type 2 are reversible — you can reverse it and correct the course, if needed.
The point: you don’t need to apply a fully data-driven approach for every decision. For irreversible decisions or with high cost of failure, yes — slow, deliberate and data-heavy processes make sense. But for reversible ones — your product judgement may be more than enough.
Train Your Product Judgement
Scientific research show that our intuition and judgement are actually subconscious forms of pattern recognition based on our previous experience. That’s why PMs who’ve spent enough time in a specific industry know their customers well. They can anticipate their behaviour, understand perceived value of the product, and sense how the market will react. That’s why developing your product judgement is essential — you want to be able to trust it later with reversible product decisions.
How can you develop it? Use your product every day. Explore edge cases. Talk to customers and your competitors. Exploit your competitors’ products regularly. Immerse yourself in the domain. Wear different hats in your team: marketing, support, copywriting, QA. The more perspectives you wear, the sharper your judgement becomes. It’s a skill you build through experience and daily practice.
Pre-PMF vs Post-PMF
You need to consider where your product is in its lifecycle. If it’s still pre-Product-Market Fit, it can be costly and even counterproductive to quantify most product decisions. At this stage, founders and PMs should lean more on product judgement. Don’t get me wrong. You should track and know perfectly key metrics of your product. But speed matters. And overanalyzing slows down. That’s why Y Combinator often advises building for yourself first. When you’re the customer, you know intrinsically if a product solves your problems or not.
However, when your product is in post-PMF lifecycle and you have more data, a growing customer base and working product, making decisions should be a mix of product judgement and data. The balance between judgment and data should reflect the risk and cost of getting a decision wrong.
The data-informed pill
Instead of being data-driven, you can say that you are data-informed. But it may sound too vague for your team, stakeholders and moreover for yourself. It would be better to find the balance between data and product judgement and then communicate to others where the balance in decision-making lies. And it’s a rather complex task, as the balance will always fluctuate and move depending on your experience, product lifecycle, market conditions and the perceived cost of a decision.