Metrics Reviews 101: The Secret Sauce of Great Products

If I could point to one of the turning points of that made Codecademy’s revenue takeoff, it was the introduction of a metrics review process to the product team.

It dramatically transformed our understanding of how our metrics worked and how to improve them.

Metrics reviews are essentially the process of trying to draw the connection between the work that you did and how that impacted your key metrics.

This was hands down the most effective thing we did as product managers and once you understand the exact relationship between your work and how your company makes money, you can never work in any other way.

Before we did Metrics Review, we tried to increase conversion to paid by:

1. Updating our pricing page
2. Updating our pricing page again

After we did Metrics Review, we started working on:

1. CRO tests the common upsell points in the product
2. Detecting and fixing broken payment features faster than hurt conversion
3. Adding more robust functionality to checkout to prevent errors
4. Rewriting all of the error codes that came back to users
5. Adding upsell points in the right contextual places
6. Allowing anonymous users to buy an account, which made paid ads work
7. Iterating on our free trial model
8. Improving our payment processing and win-back rates

You get the idea. Understanding what drives a number allows you to actually improve it much more reliably and you feel very dumb for not having done this sooner.

Why Do Metrics Reviews?

The most common objection that I hear from companies is that “we know our numbers”.

However, when you drill into what they mean by this, it means that the founders/CEO occasionally look at top-level numbers like MRR and sign-ups.

The problem with this is that most subscription products have many, many more moving parts than people think about and there can be a long delay between improving the product and seeing big revenue growth.

The biggest danger is that you misinterpret your situation based solely on your top-line revenue and take the wrong actions in your product and business.

• You could see MRR going up, which might just be momentum from the past, when in fact the product is getting worse and you don’t fix the problem.
• You could see your MRR declining, which might be large older cohorts churning, when in fact your product is getting better and you mistakenly reverse course.

Your recurring revenue is influenced by multiple factors that are hard to intuit:

1. Cohorts from the past are churning or retaining at different rates, depending on how/when they were acquired.
2. Payment processing issues come up, which might increase your churn in a way that’s unrelated to your product experience.
3. New features might have a stronger or weaker impact than anticipated impact.
4. Old features might be breaking down and causing experience issues that you’re not detecting.
5. Your traffic might be seasonal, which is impacting your conversion rates and revenue.
6. Marketing campaigns might be bringing in a different kind of user that retains or activates differently than past users

This list keeps going on…

The ultimate value of a good metrics review process is that it forces your team to understand the relationship between the work that they ship and how their numbers move.

The better that they understand this, the more likely that they can build a roadmap to move these numbers.

How to Metrics Review Meeting

One thing to clarify at this level is the type of meeting that we’re talking about. At medium & larger sized companies, it is very common to have to explain to senior leadership your overall progress against a goal.

This is not the type of meeting that I’m talking about.

Presenting in these meetings is stressful, your narrative has to make sense and you don’t want to appear to not know your numbers.

A Metrics Review meeting is one that is held between peers, maybe with a few leaders, and the whole goal is to discover what you don’t understand about your numbers.

The Structure

The typical structure for a meeting like this is:

1. Re-occurring - typically every week or other week. Often enough this stays top of mind but long enough between meetings that you can do the research you need to answer questions from the last session.
2. Small & mostly between peers - So your 3-4 product managers and VP of product. Or your paid media lead, SEO lead, email lead, and your VP of marketing.
3. Long enough to facilitate discussion - You want enough time to get into an open-ended discussion on why something is happening. Probably at least 20 minutes per presenter.

The big variables here that will change between companies are:

• How often does your team ship changes?
• How long it takes you to find data insights?

So if you ship once a month and it will take you 2 weeks to get an answer back from your Data Science team, you need more time between meetings.

If you ship weekly and you have self service analytics tools, you can do this weekly. Find the cadence that works for you.

Setting the Right Tone

Before I get into exactly how these meetings function, one point to highlight is that setting the right tone of these meetings is key.

The wrong tone is senior leaders yelling at the team leads whenever a metric goes down.  This will cause the team leads to not present ambiguous or negative trends in the product, which will make this meeting a “check the box” exercise that produces little value.

The right tone is somewhere between a thought exercise and a guided exploration of a problem.  The presenters should be able to present what they understand but also honestly discuss what they don’t understand.

The Flow of Each Meeting

In each meeting, the presenter is trying to explain to the group:

1. What happened to our core metric?
2. Why this happened? Both what we changed and what happened outside of our control.
3. What do we not understand? These are the questions to research for next time.
4. What’s coming up in our roadmap that impact this?

To give an example, if I was a product manager in charge of free user sign-ups, that might look like this.

1. What happened? Our free user sign-ups went from 5k per week to 5.9k
2. Why? Well, we launched an experiment on our homepage that showed an 11% increase in conversion. Marketing updated the free email sequence, however, there was no measurable impact. Its the February, so we don’t expect any changes in traffic to the site at this time of year.
3. What do we not understand? Well, only 80% of the sign-ups happen on the homepage, so this only explains about 400 additional users, not 900. What happened at those other sign-up points? Have we seen a change in the mix of traffic? Did our top 10 keywords from SEO change? etc.
4. What's coming up? We are running a technical SEO audit, which will result in some back-end changes that might impact SEO in 3-4 weeks. The paid media team is testing a new campaign on TikTok, which has a spillover effect on free sign-ups, this is launching on 2/12.

This leads to a prioritized list of questions that the presenter can look into and come back with answers next time.

So What Do You Do With This Information

Attempting to draw the relationship between the work that you do and how your business makes money is one of the few things that I believe 100% of companies should be doing.

That said, you need to have a few things set up before this process can work.

1. Clear Top-Level KPIs - You need to have goals that you are trying to move and a clear definition of how those numbers are defined.
2. On Product User Tracking - You need to be able to track, with reasonable reliability, the core actions that your users are taking on the product. No one’s tracking is 100% reliable, but you should be able to generally trust it.
3. The ability to build dashboards quickly -  These can either be tools like MixPanel or Amplitude, an analyst that you have on the team, or the Excel skills to download CSV reports and build charts.

Once you have these in place, you have the foundational elements in place.

Even with these in place, I would start with as small of a group as possible and at as high a level as possible.  It probably took the Codecademy team 3 months before we actually asked the right questions and looked at the right data.

Also, once you start to go down this road, I have seen companies hit the following bumps.

1. Their event tracking system is more broken than they thought
2. They are working in the wrong area to drive their goals
3. Their work has a much, much smaller impact than they thought
4. Their KPIs are being calculated incorrectly
5. Teams that should be working toward the same goal are, in fact, not.
6. A very small % of your overall roadmap is actually working towards your objectives.

While these hurt to acknowledge, you now at least know about them and can start to fix them.