Cohort analysis - 4 ways to analyze your product retention rate




Cohort analysis - 4 ways to analyze your product retention rate



We all know the importance of retention for the long term success of our products.
Retention is the key to creating a sustainable business. It shows long term engagement of your most loyal users - and that’s a strong sign of a product/market fit. It influences how much revenue will each cohort produce over time and therefore what would be the lifetime value.
Higher retention = more recurring paying users.

Retention is the key for creating a sustainable business. 

And while tweaking your onboarding process and funnels may drive immediate improvements in conversion rates (and produce instant gratification) - retention is a long term process, it often requires some heavy lifting, deeper analysis, but once it pays off - it can dramatically change the business results of your product.

In this post, I would like to share 4 useful variations of retention reports that can become very handy when analyzing the performance of a product across time.

So let's begin:

4 practical methods to analyze your product's retention rate and learn about the sustainability of your business.

1. The basics: 

Common retention reports use a certain time unit (i.e. days, weeks, months) and base the cohorts on the installation date (if it’s a mobile app) or the registration date (if it’s a web-based product). They measure the number of users that originated on a certain cohort and keep using the product across time.

The report shows how many users were "born"(started) each day/week/month and what happened to them across the next time units (how many churned and how many continued to use the product).

The average mobile app loses 58% of the users in the first 30 days (42% retention rate) and loses 75% of the users in the first 90 days (25% retention rate). These vanilla numbers are based on user app visits.

Although counting repeated visits can help you understand the engagement level of the users across time - for many products and businesses this is not enough and the retention needs to be based on a meaningful activity that is expected to be repeated by the users (for instance: making a purchase or completing a session).

Here’s a made-up example of a retention report that uses the definitions of our own startup: Missbeez - a marketplace for lifestyle services on-demand.

This retention report analyzes how many paying users (users who ordered services through the app) or each cohort, continue to use the app and order more services in the following months.
The numbers are made up of course, but I tried to place them in a way that will tell a story of a real business:

Calculating your product's retention rate - an example
Retention rate based on the number of paying users each month

The chart above presents the retention based on users who completed their bookings every month within a period of 16 months.
  • For example, in November 2020 (yes, I know it’s in the future), the product had 146 new paying users (month 0), 102 of them ordered in the next month (month 1), 88 continued to use the app on the next month (month 2), and so on. 
  • Out of the first 100 paying users of the first cohort (January 2020), 24 users made an order in month 15. Which means that the retention for this cohort, after 15 months was 15%.
We use months because a typical paying customer in our business orders once a month. The cohorts are based on the registration dates, but if your product suffers from “late conversion” (where the first significant action happens long after the download/sign-up date) you may want to set your cohorts based on the first activity date and not the sign-up date. Every entrepreneur (or a product leader) usually knows the business well enough to decide.

I’m a visual kind of guy, so I usually prefer charts over tables:

Cohorts retention rate by monthly payers

In the chart above, you can see the retention based on the number of paying users in each cohort across the months.




Now, personally speaking, I’m able to digest and analyze all of the 15 cohorts in a just few seconds... it’s my superpower (not)… but if you don’t have such superpowers - it might be a good idea to select a few cohorts just for the sake of visibility.

Here’s a cleaned-up version:

Cohorts retention rate by monthly payers

Of course, every cohort has its’ own story:
  • Some cohorts start with a higher number of users due to a certain marketing campaign or product improvements 
  • Some cohorts are influenced by holidays or seasonality
  • Some cohorts are just better than others because of external market forces (more about that in here

Because every cohort has a different starting point - using actual numbers doesn’t make it easy to identify trends and patterns, so the next thing to do is to convert the numbers into percentages:

Cohorts retention rate by monthly payers - percentages

Once the values are normalized into percentages - it’s easier to identify trends, compare different cohorts, calculate averages and more.

Here's the average retention rate:

Average retention rate (by monthly payers)

Things to look for:
  • If your number of monthly paying users (by cohort) keeps dropping - you have a leakage - that’s a bad sign (although in some businesses it might still be OK! It all depends on how hard it is to bring those users onboard - but more about that later) 
  • If the number of monthly paying users (by cohort) stabilizes in a very low point - there’s a need to dig further into the numbers: maybe there isn’t enough product/market fit, maybe the original cohort is not focused enough, maybe the product has some bugs that cause users to drop. 
  • If the number of monthly paying users (by cohort) stabilizes in a high point - you rock! But don’t start celebrating yet - there are some more numbers to examine. 

Another good exercise is to compare time ranges and see if the retention is improving across time.

Here’s a comparison between the first and second half of the year:

Average retention rate (by monthly payers) - half 1 vs. half 2

As you can see from the above, it seems like the second half of 2020 is performing much better than the first half. It might be the result of some product improvements or marketing activities - but even without exploring why it happened - the bottom line is quite positive.


2. Switching from users to revenue

There are a few problems with measuring retention based on the number of paying (or active) users each month:
  1. It doesn't measure the level of engagement of the cohorts (number of sessions, deals, volumes, revenue)
  2. It doesn’t measure the frequency of use (some will use the app once a month, some will use it every day) 
  3. It’s impossible to know if the users are using the product every month or sporadically (i.e. once every 3-4 months)

In order to further explore the behavior of the cohorts - I typically measure 2 additional retention reports:
  1. Retention rate based meaningful actions (number of sessions, orders, posts, games, etc.) every cohort produces in each month  
  2. Retention rate based on the revenue each cohort produces in each month
For Missbeez I always use the following retention trio: users, orders and revenue. The 3 of them together give a good picture of the cohorts behavior, the frequency of use, the engagement level etc. 

For the sake of keeping this post simple - let’s dive into the revenue option and see what can be learned from it.

Here’s the full retention table measuring the revenue each cohort produces across months:

Calculating retention rate by measuring cohorts revenue


And here’s the average retention based on those numbers:

Average retention rate based on cohorts revenue

As you can see from the above, in month 15 (that’s 15 months since sign-up) - the cohort produces 38% of the revenue it did in the first month. Earlier we saw that when we measured the number of users - we found that 24% of them used the product in month 15. Here, the number is quite higher: 38%.

This means that 24% of the cohort’s original users (paying users to be accurate) produces 38% of its’ first-month revenue.
This is a good indication that the retained users are becoming more engaged as time passes.

Let’s repeat the time range comparison to see if we can find a trend:

Average retention rate based on cohorts revenue - half 1 vs. half 2

As you can see from the above, as the months pass, the revenue is actually going up.
Beautiful trend.

It means that although many users churn, the remaining users are spending more money than before (The ARPPU is increasing, in a Land and Expand fashion).
It's an important sign of engagement: the retained users are becoming more engaged over time.

The Smiley retention curve

Shameless bragging anecdote ahead:
Analyzing our real cohorts at Missbeez, we often found an interesting trend where the cohort’s monthly revenue decreased (in the first few months) and then slowly went up until it surpassed the revenue it created in month 0. We call it the smiley retention rate, which is also known as a Net Negative Churn.

It’s quite unusual, but it can happen:

The smiley retention graph


OK, enough bragging, let’s move on…

Check out the best of the mobile spoon in here


3. From revenue to cohort-based revenue

Here’s another cool way to look at your cohorts and understand the impact of retention on your product’s performance (did I just use the word cool for describing a report?).
This cool layered graph (what’s with that word again?!) shows the monthly revenue broken down by the different cohorts.

It’s one of the best ways to demonstrate how retention supports the growth and the long term sustainability of the business.

Take a look:

Cohorts based revenue

Every layer in the above graph represents the cohorts I used in my previous examples, it’s the same data: the thickness of each layer is the visual representation of the retention rate of the cohort (measured in revenue).

Higher retention leads to thicker layers - and that leads to a bulk of layers supporting your monthly revenue stream, just like MRR works in subscription-based products.
In the made-up example above, the revenue grows in around 10%-12% each month (MoM) although the number of new users each month grows in just 3%-4%. The rest of the growth is supported by the revenue generated by the repeat customers (thanks to a fairly high retention rate).

As compassion - let’s see what might happen if the retention rate is very low, and those layers keep vanishing across the months:

Cohorts based revenue - the bad example

As you can see from the above example: many layers last for only 5-6 months and then they disappear.

In order to maintain some sort of growth, this example assumes that the company spends much more money on binging new users onboard than the previous one, they don’t last for long, so the budget constantly needs to grow while repeat revenue stays small. The result is that while the revenue in the previous example reached $60K without spending a lot on user acquisition - this example reaches $30K while spending much more on user acquisition.

One graph - so many insights.

Many startups increase user acquisition budgets to demonstrate growth: as a result, the revenue graph goes up, but experienced VCs can immediately identify the problems by analyzing retention rates and using this cohort-based report.

BTW: in some markets, such kind of a model might still work: if it costs close to nothing to bring new users onboard - the cohorts may lose most of their value after a few months and still be profitable.

In order to know where’s the exact sweet spot, and when does each cohort become profitable - we need to move on to the next (and last) report for today: the cohort analysis report (drums drums drums). Details below:


4. The cohort analysis 

So you want to know if a certain cohort is profitable or not.
You want to know how long does it take for each cohort to pay back the CAC (cost of acquisition). In order to do so, we will have to tweak things up in our revenue-based retention report, add the CAC and basically compare it with the LIV of each cohort.

Let’s dive in:

There are a few steps required in order to create this report:
  • First, insert a column (right before the first-month column) named CAC and enter the total cost of acquisition for each cohort. 
  • Then, replace the values in each cell from gross-revenue (that’s what we used in the previous retention table) to net-revenue. In my made-up example below I multiply all the numbers with 0.3 because I assume that 30% of each transaction is kept as the commission for the deal.  
So the modified table should look something like this:

From retention rate to the full cohort analysis

And now for the last step:
  • Duplicate your table and change the cells values from simple values to running totals, so instead of showing: {-5,000 | 1,500 | 825 | 600} - that first row should now present: {-5000 | -3500 | -2675 | -1625}. 
Here’s how it should look like:

From retention rate to the full cohort analysis

The idea is simple: each row represents a cohort: it starts with the CAC (how much money did we spend to “buy” the cohort). Then, the cohort starts to generate revenue and in each month that passes the cohort’s debt is decreasing (thanks to the net-revenue it generates) until it pays back the CAC, breaks even and becomes profitable.
Using conditional formatting makes it easy to spot those break-even points in time.

If that’s not a beautiful sight I don’t know what is. 

As you can see, with the first few cohorts it took 7 months to break-even. In January 2021 (12 months later) it took only 3 months to break-even.

This makes sense of course, as we saw earlier that the later cohorts have better retention and are generating more revenue, but the beauty with this full cohort analysis report is that it visualizes the value of each cohort in a very powerful way.

At the end of the day, we want our users to remain engaged across time. The retention rate is crucial, but it’s more important that the retained users will continue to create a revenue stream. That stream is the basis of their LTV (lifetime value) and it needs to be compared to the cost we paid to bring them onboard.

Let’s take a look at a few cohorts presented in a graph:

Cohort analysis graph

The break-even moment happens when a cohort breaks-through that sexy zero-point and rises up.
It’s like a phoenix arising from the ashes!
(forgive me… I’m getting over-excited around numbers) 
 Cohort analysis graph - the phoenix

While reaching profitability in each cohort is something investors will definitely look for - most VCs will pay attention to how fast it happens. It’s much riskier to invest in a business where cohorts become profitable after 6-12 months (the startup might die before enough cohorts reach profitability and any hiccup might lead to additional losses).

If your business doesn’t sell long term dreams and needs to demonstrate a profitable model - that model needs to not only be well structured and stable - it also needs to show how it achieves cohorts profitability quite fast.


What happens if the phoenix never arises?

Cohort analysis graph - the dead phoenix

As you can see from the above - Cohort A starts with a fairly low CAC but the retention seems so low the net revenue is barely growing. As a result, the poor phoenix never rises from the ashes…

Cohort B, on the other hand, does show some decent net revenue growth (at least in the first few months) but the CAC is so high, it's going to take forever to reach that desirable break-even point, if at all…

When reaching such situations, there are a few things to do:
  1. Make sure no one, never ever sees this horrible vision. (I can’t get it out of my head!)
  2. Focus on the CAC (more relevant for cohort B) - maybe your user acquisition strategy is wrong, maybe the wrong users are brought into the system. Is it possible (even if not right now) to bring users in half of the price? If the answer is yes - the future cohorts can be profitable. 
  3. Explore the reasons for the poor retention rate (more relevant for cohort A) - are you lacking a product/market fit? Is there a pricing problem? Product limitations? Without improved retention rates, it will be hard to build a sustainable business with such numbers. 



Summary: 

This post is about retention. As you can see, there are many ways to explore the performance of the cohorts. Almost all of them are directly driven by retention rates.

High retention rates can be translated into a solid, repeated monthly revenue, that supports both the stability of the business and its’ growth, and that’s even before improving other aspects of the product such as conversion rates, functionality, brand awareness, network effect, etc.

If you found this post interesting I would highly recommend that you read my thoughts on early adopters in here: the good, the bad, and the ugly truth about early adopters. The post talks about the risks of focusing too much on the early adopters when polishing the business model and unit economics.

Other posts you may like:

10 shades of MVP (or: how to develop a product without developing a product...)11 lessons learned while trying to become a data-driven company5 Things I Learned In The Transition From B2B to B2CHow to maintain your product momentum when you’re out of development budget


Subscribe to my newsletter and receive my occasional posts directly to your inbox!




Comments

Gil Bouhnick The Mobile Spoon
Yaron Lupo said…
Gil, these posts keep getting better and better.
Plesae, keep posting - this stuff is incredible.

(and please accept my Linkedin invite ;-))
Gil Bouhnick The Mobile Spoon
Anonymous said…
Great post. I am experiencing many of these issues as well..
Gil Bouhnick The Mobile Spoon
Product Plan said…
In a world where advertising-supported products, SaaS products with monthly renewals, and free trials are de rigueur, customers don’t have nearly as much incentive to stick around as they used to. With switching costs reduced to typing in a different URL or downloading an alternative app on their phone, consumers can be as fickle as they’d like with minimal impact on their wallets; not to mention avoiding painful personal interactions to cancel their subscriptions or the hassle of returning a physical product.