Ishant Juyal’s approach
Metric that changed:
Average storage usage among its business accounts = Total storage by accounts that are considered business accounts/ Total unique number of business accounts
First thing to do here is to understand how the metric has been calculated, how the data is coming from and whether there has been any change in however this is done. Maybe before we were calculating it with only accounts with > 0 storage and now we are calculating with all accounts. And maybe how we get data was corrupted, so the data source is not accurate. We need to verify this too.
-
Then we look at the external factors: a. Has there been any changes in rules and regulations that could have caused this? b. Is there a competition that is giving better offers and users are transferring their data to them?
-
If the data source is correct, we have to investigate things in this order: a. Is this change sudden or gradual? b. Is this change all over the companies or concentrated to some companies. For this, we can check the median and see if that has changed too. Then we can make a pivot table of all the companies with certain kind of storage and see where there is a big shift. c. Then we create user segments - based on user tier, number of team members in business, newly subscribed vs longtime businesses, region (country, continent, state etc.) d. Is there any change in the metrics that define the user behaviour (new user adding data, old users adding data, deleting data, transferring data). If there would be any changes, we will find out form here.
-
After this, we will get some insights into how the user behaviour is and based on that we will come up with a hypothesis. For example, the hypothesis could be that the #1 company with the highest storage has churned, hence the overall mean has come down.
-
Then we will validate this hypothesis.
-
Once validated, our next step would be to come up with ways to counter this. We either say that the mean is not the right metric, and maybe we should look at median. Or we can say that we want the mean to come back to its original value.
-
Based on which direction we want to go, we will have our business goal and based on that, we will create a product roadmap to achieve that goal.