From data-driven to data-informed: The importance of experimentation in product management

Note: A version of this piece was originally published by Allie O’Connell, a former Senior Product Manager at Carbon Five, a West Monroe company. It has since been updated with our latest understanding of best practices in product management—specifically, a subtle but key difference between striving to be “data-driven” versus “data-informed.” Data-informed suggests we use data as an input in making decisions, instead of data-driven, which implies that the data decides. 

Data is everywhere—but knowing how to find it and how to use it is what sets the most effective software development and product management leaders apart. In order to effectively align product success metrics with overall business goals, some of the key steps product managers can take to create a more data-informed approach include: 

• Starting with a baseline 
• Tracking the vitals 
• Defining a North Star 
• Defining and prioritizing assumptions 
• Creating and tracking experiments 
• Reviewing and restarting as needed 

Those who work in this space today may be familiar with these steps—but with that familiarity can come assumptions and analytical myths that are often held as truths. A few important points to note: 

Qualitative data is equally as important as quantitative data

Numbers are often held as hard facts, but there is no substitute to evaluating the effectiveness of your product and work without one-on-one user interviews. Qualitative data points are just as important and significant to guide both product and business decisions. Understanding qualitative data points can successfully help nurture user growth and strengthen retention. 

Your “North Star” will change as your company grows

The “North Star” data point is the core metric that defines the relationship between customer problems and the revenue the business wants to generate. If you’re a startup, it might be how many client accounts are created via user referrals. If you’re growth stage, it might be how many uploads users are contributing on a weekly basis. Keep in mind that this metric will likely change as your company grows and your product matures. It is the metric you should always be evaluating. 

Leveraging experiments to make data-informed decisions 

It's essential to strike a balance between relying on data and implementing experiments to validate assumptions and test new ideas. Instead of being overwhelmed by the vast array of tools and metrics, focus on the baseline metrics you created at the beginning. Keep a prioritized list of assumptions and new ideas, and design experiments that can inform decisions and drive improvements. The relationship between data and experiments is vital, as data helps you identify areas that need attention, while experiments enable you to test different approaches and measure the impact of your actions. 

Analyze, review and restart as needed

Being data informed is cyclical. You’ll either be analyzing, experimenting, reviewing or restarting and that’s OK. Keep evaluating your North Star. Remember your goals and evolve your metrics as your product continues to grow.