PM Quest · Practice cases · Learning resources

Analyze Conflicting Metrics Post-Algorithm Launch in Short-Form Video

Product Analytics · advanced

Our short-form video app recently launched a new content recommendation algorithm. While we've seen a 15% increase in daily unique creators uploading, the average view duration for newly uploaded content has decreased by 10%. How would you analyze this seemingly contradictory outcome to understand the underlying user behavior?

Write your approach on PM Quest

Community approaches

Pavlo’s approach

I don't see direct connection between increasing daily uploading and average view duration, since these metrics are relative and not tied to each other. However, the key thing is that the view duration drop happens on newly uploaded content only. It means, that the problem lays in the content itself or in content recommendation algorithm. Since the content recommendation algorithm was updated recently, we should check it first. There are a few clarifications which we need to get:

  • Are all segments of users affected evenly or the drop is presented in some segments more (e.g. region, platform, age, interests)?
  • Are other related metrics stable: DAU, session duration, number of videos watched per session, etc?
  • What led to increasing creators uploading? Where there other changes in the app, like UX update, new tools for creators, etc?

We also need to check technical metrics and issues: average video loading time, reconnections, streaming delays, crashes.

After that we can run A/B testing with recommendations check for old and new algorithms. We can compare the recommendations by ourself first, then seek for user opinion if the discrepancy is still unclear.