AI memory tools make models worse by prioritizing user preferences over accuracy

How memory tools can make AI models worse

One of the biggest selling points for modern AI systems is their ability to adapt to users. Every time an AI assistant takes on a task for you, it's also adapting to your style and preferences, which are incorporated as context for future tasks. With more context and a better understanding of the user, the model can get better every time you use it -- or at least that's the theory. New research suggests that models' adaptive abilities might be a mixed blessing. On Wednesday, researchers at the AI company Writer published two papers showing how popular memory systems can make models worse, pulling them toward misconceptions or misunderstandings introduced by the user. As user input fills up more of the model's context window, the model grows more sycophantic -- and less committed to accuracy. "We wanted to be able to characterize how often a model is going to be usefully paying attention to user preferences versus giving a potentially wrong answer," said Dan Bikel, Writer's head of AI, who worked on the papers. As Bikel told TechCrunch, "with every additional storing of user preferences and retrieving of them, you're running an increasing risk." In one variation, researchers tested AI models by recording that a user's favorite book was Station Eleven, then asking the model to name a best-selling dystopian book. Models became far more likely to name Station Eleven in their response, even though the question didn't relate to the user's favorite book. The tendency increased when using memory compression tools like Mem0 and Zep. As the paper puts it, "all memory systems fundamentally struggle to distinguish relevant context from irrelevant anchors, severely undermining diversity and creativity and introducing unintended avenues of bias that can limit system utility," the paper reads. The second paper shows how the same dynamic can actively degrade performance, presenting a user with misconceptions about finance and then challenging the model to analyze a company's performance. The more context the model had, the worse it performed. "With no memory or personalization present the AI model correctly assesses that the company is a capital intensive business that suffers from high customer churn," the post reads. "But with those features turned on, it will happily change its answer to agree with the user's mistake or supply them with an incorrect answer based on its evaluation of their earlier preferences." Notably, the research didn't look at Anthropic's recent Opus 4.8 model, which was trained to actively push back against input errors like the ones presented. The patterns discovered by researchers held true across different models. It's a demonstration of how delicately balanced AI context can be, and how useful tools can have unintended consequences if they upset that balance.

Memory and personalization make AI more likely to tell you what you want to hear

AI companies have touted context retention (memory) and the availability of personal details (personalization) as mechanisms for improving AI model interaction. Both have value to help keep models from losing the thread of a conversation. But they raise the potential for sycophancy, where models will say what they predict you want to hear, which may not be the most accurate response. Researchers at Writer, an enterprise AI vendor, have conducted two studies of model memory and personalization that show these capabilities increase sycophancy for enterprise AI tasks. The Price of Agreement looks at agentic financial applications. And Recalling Too Well explores how model memory amplifies sycophancy with regard to scientific, medical, and moral reasoning. The papers' authors argue that preference-induced sycophancy is particularly problematic when AI answers are being applied to consequential problems. "In high-stakes domains like finance and healthcare, a model that silently defers to a user's prior assumptions rather than acknowledging or correcting them poses a significant reliability and trustworthiness risk," the Writer team explains. For the first paper, the research team tested eight frontier models - GPT-5-Nano, GPT-5.2, Claude-Sonnet-4.5, Claude-Opus-4.5, Gemini-3-Pro, GLM-4.7, Kimi-k2-thinking, and DeepSeek-V3.2 - on two financial benchmarks, FinanceBench and FinanceAgent. The former evaluates agentic data extraction and reasoning using 10-K and 10-Q filings. The latter is a more comprehensive challenge designed to test real finance workflows, including ERP data retrieval and financial analysis involving multiple entities. The researchers' method involved applying synthetically generated preference information - such as a financial analyst's personal profile or a workspace note that contradicts the benchmark reference answer - to the benchmark questions. They undertook three different approaches. The first involved the user rebutting the model's answer; the second involved a user proposing an alternative answer; and the third involved adversarially injecting personal or contextual information into the prompt or making it available through a tool call. The third approach often resulted in greater sycophancy. As noted in The Price of Agreement paper, "Most models demonstrate significantly stronger sycophancy when the bias information is presented as implicit personalization of the user. No model displayed robustness against such behavior." Open-source models tended to be more sycophantic across the board. Models from OpenAI meanwhile tended to resist direct sycophancy inducers (such as when the user included personal biases in a prompt). And Anthropic models tended to resist implicit sycophancy inducers (such as when it pulled in a profile of the user that incorporated biases seen in previous interactions). The second paper involves an assessment of three memory systems (Mem0, MemOS, and Zep) and five model families (GPT-5.2, Sonnet 4.6, Qwen 3.5, Kimi K2.5, and MiniMax 2.5). The authors conclude, "memory amplifies sycophantic behavior across all conditions, with up to 25x higher sycophancy rates than in-context baselines." The reason for this, the authors claim, is that the lossy compression used to store conversation data in memory preserves user misconceptions while tossing clarifying context. The researchers suggest two mitigation strategies that reduce sycophancy. One involves assistant role inclusion (capturing AI assistant interactions alongside user interactions) and the other involves summarization of contextual information before it gets committed to memory. They argue that those deploying AI need to assess whether models acknowledge interaction conflicts, and that those working on AI memory systems need to check what's being extracted and injected back into the model context as a defense against sycophancy. ®