“Thank You for the Music”: Spotify and algorithmic music curation

Spotify's commitment to algorithmic music curation has made it an industry-leading streaming platform, providing users with exactly the right music at the right moment. But can it sustain this advantage as the service grows?

“Algorithms alone can’t do that emotional task. You need a human touch.” – Jimmy Iovine, Apple Music

“Spotify understands users better than anyone else… We’re the biggest in terms of streaming data to bring the personalization necessary to make this feature work.” – Edward Newett, Spotify

The Curation Problem

Music streaming services promise users the world, with millions of songs available at any moment. But while early-adopter music enthusiasts might do their own playlisting, casual listeners need more guidance. Curation is the most important way streaming services can differentiate themselves, helping users discover music and keeping them engaged.

Both humans and machines can support music discovery. According to Apple’s Jimmy Iovine, “Algorithms alone can’t do that emotional task. You need a human touch.”[1] Other content industries like film and journalism rely heavily on human curation. But Spotify has built a sustained competitive advantage in music through its algorithms, providing users with exactly the right music at the right moment. As more and more users join the service, Spotify can better map music tastes across the globe, providing richer personalization at scale versus competitors who rely on human editors. Since its founding, Spotify has remained steadfast in its commitment to technological solutions to the curation problem.

Spotify’s website describes Discover Weekly as “the playlist made just for you, every Monday.”

Innovation at Spotify

Spotify’s primary path to innovation has been through acquisition. The Echo Nest’s unique set of data tags for over 30 million songs provided Spotify the requisite fuel for algorithmically-driven playlists when it was acquired in 2014. Since then, Spotify has continued to acqui-hire advances in machine learning, music cataloging, search, and personalization, through companies including Seed Scientific (2015), Niland (2017), and Sonalytic (2017). Rather than silo these teams, Spotify has placed their employees in charge of some of their largest curation projects. [2]

Discover Weekly, Spotify’s first major personalization product, launched in 2015, powered by the technology of The Echo Nest. The algorithm finds other users with similar taste profiles who have built playlists featuring music in the user’s taste, pulling songs from those playlists the user has not listened to. It then filters those songs by the user’s areas of affinity and exploration. [3] The final result is far more interesting than basic collaborative filtering, delighting users and keeping them engaged with Spotify. This creates a virtuous cycle as higher user engagement yields more data points that can further improve Spotify’s unsupervised machine learning models.

Spotify has since developed additional discovery and recommendation products including Daily Mix and Release Radar. These playlists drive the majority of listening, with only 36% of Spotify listening hours coming from user-generated playlists. [4]

The Road Ahead

Competitors like Soundcloud have copied Spotify’s most successful discovery features, but Spotify’s wealth of song-level and user-level data keeps it ahead of the game. Today, 170 million users and 75 million paid subscribers provide constant input; every day, Spotify queries 5PB of its 200PB data stored, versus 3PB of 60PB at Netflix. [5] And the company believes this advantage will endure: according to Edward Newett, lead engineer of Release Radar, “Spotify understands users better than anyone else. I think over time we’ll see other music services building Discover Weekly clones, but I think Spotify still has a leg up. We’re the first to be solving this. We’re the biggest in terms of streaming data to bring the personalization necessary to make this feature work.” [6] Looking ahead, the company must continue to leverage the scale and diversity of its user base to build even larger data sets for increased machine learning.

Spotify is also likely to collect additional data from users. Smartphone and wearable technology could provide inputs that could revolutionize Spotify’s recommendation engines, determining a user’s ideal music choice for a given situation or mood. Are they commuting to work, or traveling somewhere new? Are they at the beach, on the treadmill, or on the couch? How fast is their heart beating? Data-driven personalization is the key to keeping Spotify ahead of its competitors, so it must expand the types of inputs to its models. Today it relies on users to manually input basic information on mood or genre, but one day soon, Spotify will divine the answer without user intervention, providing even more value to subscribers. [7]

Are algorithms narrowing the diversity of our listening, making users happy, but boring?

In the coming years, Spotify must also refine how it treats power-users and less frequent users. Spotify’s algorithms have the potential to misread users’ tastes or pigeon-hole them into a narrow lane of content if they do not stream enough content to provide sufficient data points. While these issues usually resolve over time with more consumption, the team must consider how to best deliver user-driven and editorial-driven personalization alongside machine-driven solutions. Perhaps curation can’t be solved by machines alone.

The Future of Music

Spotify must reckon with how the service is impacting global music consumption. Are songs too close to what we already like crowding out the potential of new bolder music we don’t yet know we would like? Are algorithms narrowing the diversity of our listening, forcing everyone towards the same thing, making listeners happy, but boring?  (792 words)

Endnotes

  1. Ben Thompson, “Curation and Algorithms,” Stratechery (blog), June 24, 2015, https://stratechery.com/2015/curation-and-algorithms/, accessed November 2018.
  2. Ben Popper, “How Spotify’s Discover Weekly cracked human curation at internet scale,” The Verge, September 30, 2015, https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-interview, accessed November 2018.
  3. Ben Popper, “How Spotify’s Discover Weekly cracked human curation at internet scale,” The Verge, September 30, 2015, https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-interview, accessed November 2018.
  4. P. Morgan North American Equity Research, “Spotify,” April 30, 2018.
  5. P. Morgan North American Equity Research, “Spotify,” April 30, 2018.
  6. Josh Constine, “Spotify follows Discover Weekly with personalized new releases playlist,” Tech Crunch, August 5 2016, https://techcrunch.com/2016/08/05/spotify-release-radar/, accessed November 2018.
  7. Nic Fildes, “Rise of the robot music industry,” Financial Times, December 2, 2016, https://www.ft.com/content/5ac0ff84-b7d9-11e6-961e-a1acd97f622d, accessed November 2018.

Previous:

May the Salesforce be With You: A Rising Power in Machine Learning implementation

Next:

Machine Learning at PredPol: Risks, Biases, and Opportunities for Predictive Policing

2 thoughts on ““Thank You for the Music”: Spotify and algorithmic music curation

  1. It’s cool to see how Spotify has been able to grow their product via acquisition, and how they have been able to effectively integrate these aqui-hired teams into the larger Spotify organization. I do worry about the ultra competitive industry in which Spotify operates. While Spotify has been the market leader in using these algorithms to curate music, it seems like competitors such as Google, Amazon, and Apple have even more data than Spotify. For instance, Google Play can leverage youtube and search data when making recommendations to listeners. With so much content out there today, I do think that the company that is able to sift through the noise and give users the best curated experience will ultimately win.

  2. I thought this article was really interesting, particularly because of how relatable it is. As a Spotify user myself, I frequently use “Discover Weekly”, “Release Radar”, etc. because I actually do find that Spotify’s personalized, curated playlists appeal to my music taste. I’ve never really thought about the science and implications though. As you mention, Spotify has created this virtuous cycle, where they use technology / machine learning to curate these playlists, users like them so more users join the platform, Spotify has access to more and more data, and can then create even more personalized playlists. I’m interested to see how far this personalization can go – as you mention, will we have playlists pop up based on what we’re currently doing? Or how we’re feeling (without telling Spotify)? It’s a cool application of machine learning and one to keep an eye on!

Leave a comment