Spotify’s new AI DJ is bringing the new flavor to your ear

“What’s going on? I’m X, and from this moment on, I’m gonna be your own personal AI DJ on Spotify. Let’s go.” —Spotify’s new AI DJ, by way of a greeting

The feature, which debuted on February 22, is fueled by years of Spotify machine learning and user listening data—think: The songs you played on repeat in 2018, or the new genre you’ve been into lately—and generative AI from Sonantic, the voice AI company Spotify acquired last summer for a reported ~$95 million.

In the time between the AI DJ’s release and the beginning of March, it saw a notable number of return users: “On days when users tune in, fans spend 25% of their listening time with DJ,” Spotify stated in a blog post. The post goes on to say that more than half of first-time listeners used the DJ again the following day.

Spotify has leaned into personalization tech for years, but the past six months—since the Sonantic acquisition—allowed for the generative AI piece of the puzzle and made the feature possible as it works today, according to Zaid Sultan, Spotify’s vice president of personalization.

“From then on, we had that extra magical piece—that was the super realistic but also emotive voice,” Sultan told us. He added: “The last six months are when the generative AI piece has, maybe, been a more prominent part of the story.”

Building the AI DJ

The team for Spotify’s AI DJ was made up of more than 10 people, Sultan told us, though Spotify declined to give an exact number. It included departments like personalization; a writers’ room of editors, data curators, and scriptwriters; and Sonantic’s own voice AI team.

The AI DJ’s voice itself was modeled after the voice of Xavier “X” Jernigan, a full-time Spotify employee who serves as the company’s head of cultural partnerships. Jernigan, who had music industry experience working at Def Jam and Sony Music—plus a stint as Sean “Diddy” Combs’s personal assistant—was tapped to be the feature’s voice model after appearing as a Spotify podcast host.

“The [machine learning] model we’re using came from the Sonantic acquisition,” Sultan said, adding, “Their focus that they have is on the quality of the output, especially on the emotions. So what was cool about this process is…They are able to work closely with X in the studio so that the input into the model is really high-quality and is also mapped to emotions. So they can find when X tends to breathe [and] what does he sound like when he’s excited.”

Jernigan carried a notebook throughout the day, writing down typical phrases he used to greet people or refer to music—for instance, he noticed he’d often use words like “bops,” “jams,” or “bangers,” but not “tunes.” That kind of information, as well as Jernigan’s inflection, went into training the model to develop an AI-produced version of his voice.

An outcome that Sultan said the company didn’t anticipate: Sonantic’s voice AI working too well.

“Once we tried [doing] user studies on the early prototypes of the DJ, the voice was so realistic that if you don’t know what it is…your mind doesn’t tell you this is an AI—your mind tells you, ‘They must've hired a person who recorded a bunch of things—[this is] a radio, or it’s a station…a product I’ve already seen,’” Sultan said.

To help convey the level of personalization, Sultan and his team had to try to figure out how to “onboard” users, and they had just one chance—one brief intro snippet of audio—to make their mark. The team conducted user studies and then decided on the intro snippet’s length, language, and more.

“Unless you understand this is an AI just for you, you don’t realize how personalized it is,” Sultan said, adding, “You don’t realize it can bring you your [Spotify] Wrapped from two years ago; you don’t understand it can give you an artist that you will love.”