PM Quest · Practice cases · Learning resources

Amazon Reviews Engagement Spike with Conversion Drop Analysis

Root Cause Analysis · intermediate

Amazon has observed an unexpected result from a recent A/B test: the engagement on product reviews has spiked by 30%, but the conversion rate for products with high-rated reviews has inexplicably fallen by 10%. This seems contradictory, as higher engagement usually correlates with higher conversions. Your task is to diagnose the root cause of this anomaly. Do not suggest product changes yet; focus purely on analysis of existing data.

Write your approach on PM Quest

Community approaches

Dhruv’s approach

STEP 1: CHECKING IF THE DATA IS RIGHT

  • Are we sure analytics is not broken anywhere in the tracking?
  • Are we sure the meaning of a conversion has not changed?
  • Over what timeline are we comparing the 10% drop? Is it the same timeline since the A/B test began?

STEP 2: ASKING CLARIFYING QUESTIONS

  • What exactly do we mean by conversion rate? Is it the buy ratio, or the add-to-cart conversion once a product is clicked?
  • I am guessing the product reviews we are counting are from folks who actually purchased the product. Or are we also counting questions asked in the comment box?
  • Have we changed any logic for calculating the rating of a product?
  • What exact changes did we make to test during the A/B? Is it a flow change or a UI/UX change?
  • Out of acquisition, activation and retention, has any of them changed?

STEP 3: SEGMENTING USERS BASIS FACTORS We segment on the basis of gender, product type, product value and buying patterns.

  • First, under each category, we check whether the increase in reviews and the drop in conversion are common to a particular cohort. We compare this against the B set.
  • If there is no common cohort, the question changes to identifying the drop only.
  • If we do find a common cohort where both changes are happening, we deep-dive into the next step.

STEP 4: GUESSING EXTERNAL AND INTERNAL FACTORS For that common cohort, we segment the factors into external and internal.

Internal factors

  • New user acquisition has fallen, because of which there is a drop in high-rated products, since new users would go for high-rated products for trust and safety.
  • The user set we chose in the A group has a lower conversion rate by default, which means the user segmentation was not done correctly.
  • We pushed a change in the buy-out flow which has caused a drop. To verify, we want to understand whether there is a conversion drop anywhere else in that flow too.
  • Does our A set have more folks who have recently bought a product, compared to the B set? If yes, then people who have already purchased tend to give a review and not buy again.
  • Have we made changes to the UI of the product page that improved the visibility of the review option while reducing the visibility of the buy now button?

External factors

  • Has there been recent negative PR about the product, which has led to increased negative comments and reduced conversion?

STEP 5: DECIDING A FEW HYPOTHESES The hypothesis should be financially feasible and logically sound.

H1: We made a UI change because of which the visibility of the buy now button reduced. H2: The user set we chose for the A test was biased, due to a high concentration of recently-bought buyers. H3: Our tracking system for the A test is broken somewhere, because of which the numbers are incorrect.