Can AI simulate the experience of fumbling through a conversation spoken in a language one only half knows?
A new feature from Duolingo lets subscribers practice their language skills through AI-powered video calls with one of the company’s cartoon characters, Lily. The OpenAI-powered tool is designed to immerse learners in “spontaneous, realistic conversations” in order to get hands-on chatting experience.
The feature was initially only available in Spanish, French, and English, but the company added German, Italian, and Portuguese soon after.
“The idea here is that speaking is a muscle,” Duolingo Principal Product Manager Zan Gilani told Tech Brew. “Up until now, finding access to native speakers has been very hard for most people…And then even when you do have access to them, it’s very nerve-racking. Often, you feel kind of embarrassed and shy.”
It’s just the latest way the gamified language learning platform has been tapping large language models (LLMs) through a longstanding partnership with OpenAI. The company is also using GPT-4 to develop its learning content and power other features like DuoRadio, a longform listening feature, and another set of mini-games rolled out this week called Adventures, which places players in common travel scenarios with Duolingo’s characters, according to Gilani.
A difficult conversation: But the process of molding the AI into a full-fledged video conversationalist proved to be a bit more involved than some of those other undertakings, according to Gilani. The team spent around a year “iterating and tweaking,” and adding guardrails to ensure that Lily could hold forth at different skill levels in an environment where the company doesn’t have full control over what the AI is going to say.
There was also the matter of making her more fun than a simple multilingual Alexa clone, Gilani said.
“A lot of these types of experiences will demo super well, but after the third and the fourth, the fourth and the fifth call, you’re just like, ‘I’m talking to a bot,’” Gilani said. “We’re really trying to make her feel human. She should not feel like an assistant. She shouldn’t feel like your smart speaker.”
Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
A range of experiments: Generative AI is perhaps easier to implement for something like language education, where the factual content doesn’t matter as much as correct syntax and grammar.
“No one’s coming to Lily to get medical advice or something like that,” Gilani said.
Duolingo’s team thinks about AI across three broad buckets, according to Gilani. One is expediting the production of content; another is adding features that would otherwise be too expensive to produce, like DuoRadio and Adventures; the third is enabling tasks that wouldn’t have been possible at all without the tech, which includes video calling.
There have also been experiments where AI was not a good fitf. For instance, Gilani said, the team tested a feature that would clone a user’s voice and create audio of them speaking in a different language as a form of motivation. “Ultimately, we were like, ‘Hey, this is, like, a bit too weird and potentially creepy,’” Gilani said.
Language barriers: Another challenge is the quality of LLMs across different languages, Gilani said. The English video calling is the “highest-quality” experience, followed by the other languages being rolled out this week. “She’s just a little bit less sassy when it comes to speaking in those [non-English] languages,” he said.
Other languages, like Japanese, Korean, and Chinese, will require extra custom instructions, he added.
As the feature rolls out, Duolingo will be watching how users engage with it, and Gilani said the hope is to eventually make conversations longer and more expressive with more personalized memory recall.