This blog is based on a Thomson Reuters pendomonium session, measuring the impact of your incredible analytics work.
“How do you measure and demonstrate the value of product analytics?”
This is a question that is of interest to many at Pendomonium 2023, including Thomson Reuters senior director of product analytics, Drew Harden. Nobody reached a solid conclusion, and that was the end of it. But then, Harden revealed the answer to this problem in his own organization and returned in 2024 to share it.
Measure the impact of your product analysis
Thomson Reuters invests heavily in product analysis: they have 30+ analysts managing pendo data from 150+ applications.
And because traditional approaches to measuring the impact of analytics — like calculating work hours saved — only capture a small slice of the impact, Harden found a more holistic approach: Count the number of product decisions influenced.
Step 1: Define success as an influenced product decision
Harden rocks through several different ways to measure “success” for product analytics – from the entire value or percentage of projects supported, to labor hours saved, and even productivity measured in terms of sales.
Spoiler alert: There isn’t any of them is satisfactory. Delivering a measurable dollar amount, hours saved, or profitability was never their team’s goal. Therefore, they shifted their focus to more tangible metrics: The number of product decisions their team influences.
Harden defines “influencing decisions” as recommending actions to product teams. This means moving away from simply providing data, and instead providing actionable insights. In 2023, Their team set a goal to influence 250 product decisions.
Step 2: Track the number of decisions you influence
Harden thought that influencing a decision meant recommending a course of action. Rather than pulling data (like NPS detractor responses) for their product managers (PMs), teams want to support projects and answer strategic questions, like:
- What steps should you take to improve your NPS score?
- How can you increase customer retention?
- Who are the customers at risk now? What about in the future?
To calculate how many product decisions they chose, they added items to their project management system, Azure DevOps (ADO).
Next, they recommend actions to engage with PMS. This can be had super nuanced, so they start by considering the behavior they want to encourage. Some guidelines include:
- The decision to do nothing still counts.
- One project can influence many decisions.
- Dashboards can influence many decisions over time.
- It’s okay if the partner team doesn’t act on their decisions.
Step 3: Involve and manage partner teams
Thomson Reuters is a large organization. Product analytics teams care about the value of their work – but how do they get everyone if not to care?
One way they accomplished this was by building a Pendo application form. Other teams often engage with these forms, and help them pivot to say, “I’d like to see the complaints from my NPS detractors,” to “I have a development decision coming up, and I can make a better decision if I know what top 10 features are. “
Although dollars are not the main value indicator, they leave room for this field in the application. This helps us understand what and when PMs will launch their projects, and how it connects to overall goals or revenue.
Here’s the cool part: Everyone loves this. They shifted from being treated as mere Pendo administrators to internal business partners. Product teams can now put more thought into their development, and their product analysts can free up time typically spent on low-impact projects for higher-value ones. The caliber of their projects immediately rose.
Step 4: Show results frequently
They’ve done all this work – now, it’s time to show their impact on the wider Thomson Reuters organization.
There are four main ways they show their team results:
- Bite-sized feedback in quarterly business reviews (QBRS). When they redesigned the tax product, they did a deep dive on a particular feature, revealed that it wasn’t used much, and recommended that they could build a sunset.
- Email summary to show how analytics can help PMS. I’ll pick the top six to eight decisions, give a few shoutouts to the unsung heroes, and highlight how they helped the product understand if they could sunset these features based on low usage data.
- Stakeholder meetings to review wins and brainstorm. They use this decision framework to talk with senior leaders, brainstorm, and list all the decisions they support their team.
- Product Analysis Report. They write, in detail, and offer a searchable library that users can explore to understand what they do in various product areas. Here’s a preview of our tempalte:
Step 5: Iterate and evolve
Our approach to measuring the value of analytics work exceeded expectations: in 2023, they influenced 341 decisions—136% higher from their original goal. But the results of this don’t end there: Thomson Reuters’ UX research team also adopted this decision framework for their CXO program.
Moving forward, they are focused on scaling product analytics impact in two ways: incorporating AI into usage forecasting and retention modeling, and building deeper partnerships with product teams.
Prefer to watch? Catch Thomson Reuters’ recorded pendomonium session instead.
Gaming Hub
A gaming hub can refer to a central platform or space dedicated to gaming, where players can access games, interact with other gamers, and enjoy related content.