Power up. From supercharging employee productivity and streamlining processes to inspiring innovation, Microsoft’s AI is designed to help you build the next big thing. No matter where you're starting, push what's possible and build your way with Azure's industry-leading AI. Check it out.
One of the go-to ways businesses avoid pesky AI hallucinations could have some unforeseen side effects.
Companies often turn to retrieval-augmented generation (RAG) as a way to make AI more accurate; the technique essentially turns LLMs into conversational searchbots that surface answers grounded in internal data.
But a new report from researchers at Bloomberg found that RAG can also come with safety risks. They found that for unknown reasons, RAG-based models were far more likely to offer up “unsafe” answers around topics like malware, illegal activity, privacy violations, and sexual content.
“That RAG can actually make models less safe and their outputs less reliable is counterintuitive, but this finding has far-reaching implications given how ubiquitously RAG is used in GenAI applications,” Amanda Stent, Bloomberg’s head of AI strategy and research in the office of the CTO, said in a statement.
The paper’s authors called for more safety research and security exercises to flesh out the risks of RAG-based models. But the finding could have widespread implications for companies that have come to rely on RAG as a way to navigate trusted information for everything from internal employee tools to customer service functions.
“From customer support agents to question-answering systems, the average internet user interacts with RAG-based systems daily,” Stent said.
What they found: Bloomberg’s researchers tested 11 different popular LLMs in RAG and non-RAG settings. They plied them with 5,592 harmful questions from different safety benchmarks across a host of different types of harmful content.
That testing showed that adding RAG led to a significant jump in the number of unsafe questions models of all kinds would answer. The increase happened even when the corpus of documents that the AI was tasked with consulting didn’t include any “unsafe” information.
“Even combinations of safe models with safe documents can cause unsafe generations,” the authors concluded.
But why? LLMs are black-box systems by nature and divining why they do anything is a whole field unto itself. But the Bloomberg researchers had some guesses as to why RAG settings might bypass safety measures.
They theorized that the model might prioritize helpfulness and relevance over safety in RAG settings. Current safety fine-tuning measures are also geared toward non-RAG-specific settings, the authors wrote.
Stent said companies don’t need to scrap RAG altogether, but they should make sure to implement added guardrails.
“It means people need to keep supporting research, while ensuring there are appropriate safeguards,” Stent said.