Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy. Learn more
Generating voices that are not only humanlike and nuanced but diverse continues to be a struggle in conversational AI.
At the end of the day, people want to hear voices that sound like them or are at least natural, not just the 20th-century American broadcast standard.
Startup Rime is tackling this challenge with Arcana text-to-speech (TTS), a new spoken language model that can quickly generate “infinite” new voices of varying genders, ages, demographics and languages just based on a simple text description of intended characteristics.
The model has helped boost customer sales — for the likes of Domino’s and Wingstop — by 15%.
“It’s one thing to have a really high-quality, life-like, real person-sounding model,” Lily Clifford, Rime CEO and co-founder, told VentureBeat. “It’s another to have a model that can not just create one voice, but infinite variability of voices along demographic lines.”
A voice model that ‘acts human’
Rime’s multimodal and autoregressive TTS model was trained on natural conversations with real people (as opposed to voice actors). Users simply type in a text prompt description of a voice with desired demographic characteristics and language.
For instance: ‘I want a 30 year old female who lives in California and is into software,’ or ‘Give me an Australian man’s voice.’
“Every time you do that, you’re going to get a different voice,” said Clifford.
Rime’s Mist v2 TTS model was built for high-volume, business-critical applications, allowing enterprises to craft unique voices for their business needs. “The customer hears a voice that allows for a natural, dynamic conversation without needing a human agent,” said Clifford.
For those looking for out-of-the-box options, meanwhile, Rime offers eight flagship speakers with unique characteristics:
- Luna (female, chill but excitable, Gen-Z optimist)
- Celeste (female, warm, laid-back, fun-loving)
- Orion (male, older, African-American, happy)
- Ursa (male, 20 years old, encyclopedic knowledge of 2000s emo music)
- Astra (female, young, wide-eyed)
- Esther (female, older, Chinese American, loving)
- Estelle (female, middle-aged, African-American, sounds so sweet)
- Andromeda (female, young, breathy, yoga vibes)
The model has the ability to switch between languages, and can whisper, be sarcastic and even mocking. Arcana can also insert laughter into speech when given the token . This can return varied, realistic outputs, from “a small chuckle to a big guffaw,” Rime says. The model can also interpret , and even correctly, even though it wasn’t explicitly trained to do so.
“It infers emotion from context,” Rime writes in a technical paper. “It laughs, sighs, hums, audibly breathes and makes subtle mouth noises. It says ‘um’ and other disfluencies naturally. It has emergent behaviors we are still discovering. In short, it acts human.”
Capturing natural conversations
Rime’s model generates audio tokens that are decoded into speech using a codec-based approach, which Rime says provides for “faster-than-real-time synthesis.” At launch, time to first audio was 250 milliseconds and public cloud latency was roughly 400 milliseconds.
Arcana was trained in three stages:
- Pre-training: Rime used open-source large language models (LLMs) as a backbone and pre-trained on a large group of text-audio pairs to help Arcana learn general linguistic and acoustic patterns.
- Supervised fine-tuning with a “massive” proprietary dataset.
- Speaker-specific fine-tuning: Rime identified the speakers it found “most exemplary” among its dataset, conversations and reliability.
Rime’s data incorporates sociolinguistic conversation techniques (factoring in social context like class, gender, location), idiolect (individual speech habits) and paralinguistic nuances (non-verbal aspects of communication that go along with speech).
The model was also trained on accent subtleties, filler words (those subconscious ‘uhs’ and ‘ums’) as well as pauses, prosodic stress patterns (intonation, timing, stressing of certain syllables) and multilingual code-switching (when multilingual speakers switch back and forth between languages).
The company has taken a unique approach to collecting all this data. Clifford explained that, typically, model builders will gather snippets from voice actors, then create a model to reproduce the characteristics of that person’s voice based on text input. Or, they’ll scrape audiobook data.
“Our approach was very different,” she explained. “It was, ‘How do we create the world’s largest proprietary data set of conversational speech?’”
To do so, Rime built its own recording studio in a basement in San Francisco and spent several months recruiting people off Craigslist, through word-of-mouth, or just causally gathered themselves and friends and family. Rather than scripted conversations, they recorded natural conversations and chitchat.
They then annotated voices with detailed metadata, encoding gender, age, dialect, speech affect and language. This has allowed Rime to achieve 98 to 100% accuracy.
Clifford noted that they are constantly augmenting this dataset.
“How do we get it to sound personal? You’re never going to get there if you’re just using voice actors,” she said. “We did the insanely hard thing of collecting really naturalistic data. The huge secret sauce of Rime is that these aren’t actors. These are real people.”
A ‘personalization harness’ that creates bespoke voices
Rime intends to give customers the ability to find voices that will work best for their application. They built a “personalization harness” tool to allow users to do A/B testing with various voices. After a given interaction, the API reports back to Rime, which provides an analytics dashboard identifying the best-performing voices based on success metrics.
Of course, customers have different definitions of what constitutes a successful call. In food service, that might be upselling an order of fries or extra wings.
“The goal for us is how do we create an application that makes it easy for our customers to run those experiments themselves?,” said Clifford. “Because our customers aren’t voice casting directors, neither are we. The challenge becomes how to make that personalization analytics layer really intuitive.”
Another KPI customers are maximizing for is the caller’s willingness to talk to the AI. They’ve found that, when switching to Rime, callers are 4X more likely to talk to the bot.
“For the first time ever, people are like, ‘No, you don’t need to transfer me. I’m perfectly willing to talk to you,’” said Clifford. “Or, when they’re transferred, they say ‘Thank you.’” (20%, in fact, are cordial when ending conversations with a bot).
Powering 100 million calls a month
Rime counts among its customers Domino’s, Wingstop, Converse Now and Ylopo. They do a lot of work with large contact centers, enterprise developers building interactive voice response (IVR) systems and telecoms, Clifford noted.
“When we switched to Rime we saw an immediate double-digit improvement in the likelihood of our calls succeeding,” said Akshay Kayastha, director of engineering at ConverseNow. “Working with Rime means we solve a ton of the last-mile problems that come up in shipping a high-impact application.”
Ylopo CPO Ge Juefeng noted that, for his company’s high-volume outbound application, they need to build immediate trust with the consumer. “We tested every model on the market and found that Rime’s voices converted customers at the highest rate,” he reported.
Rime is already helping power close to 100 million phone calls a month, said Clifford. “If you call Domino’s or Wingstop, there’s an 80 to 90% chance that you hear a Rime voice,” she said.
Looking ahead, Rime will push more into on-premises offerings to support low latency. In fact, they anticipate that, by the end of 2025, 90% of their volume will be on-prem. “The reason for that is you’re never going to be as fast if you’re running these models in the cloud,” said Clifford.
Also, Rime continues to fine-tune its models to address other linguistic challenges. For instance, phrases the model has never encountered, like Domino’s tongue-tying “Meatza ExtravaganZZa.” As Clifford noted, even if a voice is personalized, natural and responds in real time, it’s going to fail if it can’t handle a company’s unique needs.
“There are still a lot of problems that our competitors see as last-mile problems, but that our customers see as first-mile problems,” said Clifford.
