Investigating Dip in Track Completions for New Spotify Premium Converts

Pavlo’s approach

First of all, I would check if track suggesting algorithms were changed recently. If so, then we should check how could it impact a quality of suggesting. Second, I would check technical indicators, like connection delays, streaming quality, number of dropped connections, etc. Third, I would check changes in users' behaviour before and during listening, there may be some patterns which could help to understand the source of the issue.

Vanshika’s approach

I'll start with asking some clarifying questions.

1. What does track completion means ? It includes songs & podcast both or not ?
2. how's this metric getting calculated ? are other metrics in the user journey like DAU also affected ?
3. Patterns ? Users : New users or renewals too ? ; Geography : overall or limited to a region ? ; Device : Android, IOS or both ?
4. What’s the pattern / metric behaving for old users with premium subscription ?
5. Are there any changes in the UI/ Recommendation engine after premium subscription starts ?

Let's say the answers are :

1. Track completion means: User listens to ≥90% of the song duration & Skips before that are counted as non-completions. Consider Songs only here
2. Total number of tracks completed per day. No other metric like DAU is affected here.
3. Only for new users , overall, both devices
4. No similar dip
5. Yes, Users see more surfaced recommendations. No major visual redesign, but higher exposure to recommendations during playback.

Based on this information, will form 2-3 hypothesis & prioritze one out of them :

1. Access to premium membership makes users skip the song before it gets completed as now the recommendation button/options are more easily accessible in UI than before.
2. Recommendation model is not caliberated enough causing users to try new songs but could not get hooked to it.
3. Change is user behaviour, they have become more selective now as ad blockers are not available now.

I’d test the UI exposure hypothesis first because it’s fastest to validate. I’d look at the increase in recommendation clicks during playback, and more importantly, whether tracks reached via those clicks have lower completion rates than tracks that continue playing naturally.

I’d keep behavior shift and model calibration as secondary hypotheses and validate them if UI interaction doesn’t explain most of the drop.

Once validated, will brainstorm on possible solutions for this.

Ishant Juyal’s approach

The question says that there's a dip in daily track completions for all users, but especially for users who have recently converted from a free trial.

First step: Analyse different segments of users

1. Divide into cohorts by device type - Web app, Mac App, Mobile App. For simplicity, we will assume Mobile app is showing issue.
2. Divide further based on OS - iOS and Android to see if this is platform dependent.
3. Further divide cohorts based on App version to see if the problem is occurring only for users on certain new version.

For this question, we will assume the issue is with all the devices, all OS and all versions.

Second Step: Come up with a list of hypothesis on why this might be happening

1. Users are willingly not playing the whole songs a. They don't like the songs being recommended to them b. They like the song but don't want to listen to them

2. Users are not willingly skipping songs a. App is crashing b. Song gets skipped automatically

Third step: Analyse user behaviour

1. Classify users based on time spent as a premium subscriber to see if new users are more likely to skip songs in between, and what's the threshold?
2. Look at the user session recordings and events data to see what the user is doing when they change the song - Are they clicking on next button, selecting new song from the playlist, searching for songs in the search bar, closing the app?
3. Analyse the change in user behaviour for each of these actions in between songs

Fourth Step: Qualitative Analysis

1. Look at user reviews on app store, play store, other platforms
2. Look at user complaints if any regarding any change on the app.
3. Talk to a few users if possible

Fifth step

1. Run small experiments and A/B tests based on what you learn

Aryan’s approach

okay, so we are having a problem of users not completing tracks completely, and it might be because that the hook part of the song couldve ended long before the ending, and ppls generally dont listen after that, how we can improve this is by advertising our feature of excellent mixing and blending of tracks one after another, and provide an immersive experience that happens only during the switch of the song, and that happens at the end . this could potentially lead to ppls trying the feature hence improving the daily track completions.

Preathi’s approach

Problem: Considering this problem, it seems like due to a feature which doesn't exists in free trial. Like Offline playback which users can download as much as songs they want to hear and listen offline. And Unlimited skips is exclusive for premium users, so the tracks are not forced to listen, whereas free users had to listen the track fully, as they don't have the many skips. On demand selection, where the tracks are selected by itself for users who were in free trial, but premium users can select any music, album etc. without forced shuffle. So these should be the problem for this 8% decline in track completions. Or may be the recommendation quality of tracks are poor. Solution: 1. Show context "You can't play this song for the next 30 mins if you skip now" , this prevents the number skips happening as we have unlimited skip in premium trials. 2. Small card with question "why you skipping this song?" a. I want to listen other songs b. I don't like this song - this we can evaluate and ensure why users are not completing the track fully. If there is more number of users opted "b" then it is a recommendation problem, so we will improve the recommendation system by giving quality tracks based on the songs they listen repeatedly, genre, author, released duration including album, podcast etc. Success metrics: North star metric - Daily Track completion rate, L1 - No. of skips in a session per user, L2 - Where people are exactly skipping the track, (beginning, mid, end), this tells about the interest in the track they hearing , Percentage of users choose option a or b.

ahmad’s approach

Phase 1: Confirm & Scope — Verify the data is accurate and identify which user segments are affected. This prevents wasting time on false alarms or misinterpreted metrics. Phase 2: Behavioral Analysis — Test competing hypotheses: Are users listening less overall? Skipping more? Experiencing friction with Premium features? Not discovering what they paid for? Each hypothesis points to different root causes. Phase 3: Root Cause Investigation — Check technical deployments, product changes, feature flags, and qualitative user feedback in parallel. Most issues fall into a few categories: bugs in ad-serving, onboarding flow problems, algorithm changes, or feature gating errors. Phase 4-5: Diagnosis — Use cohort analysis to isolate the problem. Compare trial converters to existing Premium users, free users, and non-converted trial users. Look at metrics like skip rate, session duration, ads heard, and feature usage to pinpoint what changed. Phase 6-7: Resolution — Implement the fix (usually a hotfix or rollback), monitor closely, and communicate with affected users. Most issues can be resolved within 24-48 hours. Phase 8: Prevention — Implement monitoring, alerting, and QA improvements to prevent recurrence.

Yuvraj Singh’s approach

by investigating the new users' daily activity to set there algo and then suggest them some new recommendations according to thier taste and time at which they listen most of thier track

Trupti’s approach

As PM for Spotify Premium, I’d treat this as an early activation issue, not a broad engagement decline, since the 8% dip is limited to recent free → premium converts.

Step 1: Validate the signal First, I’d sanity-check the metric: Confirm track completion definition (100% vs threshold-based). Rule out logging bugs, experiment rollouts, or app releases. Slice by platform (iOS/Android/Web) and geo to ensure this isn’t infra-specific.

If the data holds, I move forward.

Step 2: Segment to locate where the drop happens I’d break new premium users by: Time since conversion (Day 0–1 vs Day 2–7). Listening context (algorithmic playlists, albums, search). Skip timing (first 30s vs late-track). A sharp drop in Day 0–2 or algorithmic playlists would point to onboarding or recommendation issues.

Step 3: Hypothesis-driven investigation Key hypotheses: Expectation mismatch – Premium users expect better recommendations immediately; poor sequencing leads to early skips. Onboarding friction – Too many prompts (downloads, quality, playlists) interrupt listening flow. Behavior normalization – Trial users passively completed tracks to avoid ads; premium usage is more intentional, increasing skips. Playback regressions – Higher bitrate or downloads causing buffering in weak networks.

I’d validate these via skip-rate distribution, completion curves by second, buffering events, and first-session vs later-session behavior.

Step 4: Qual + Quant I’d pair data with: Recent app store reviews mentioning “new” or “premium”. Support tickets on playback or recommendations.

Step 5: Test fixes Run focused experiments: Reduce onboarding prompts in first 24h. Bias early recommendations toward known free-era preferences. Delay feature education until after stable listening. Auto-adjust audio quality on poor networks.

Final check: If listening time and retention are stable, this may be a metric interpretation issue, not a business one. Completion rate alone may no longer represent premium value.