Healthcare AI vulnerabilities expose Doctronic's prescription bot to easy manipulation

AI doctor's assistant swayed to change scrips - researchers

A healthcare AI with the power to manage prescriptions is rather open to mind-altering suggestions, according to security experts. Redteamers at AI security firm Mindgard reported on Tuesday that it took relatively little work for them to get a healthcare AI from Doctronic not only to spill its system prompts, but to let them make modifications too. Wanna make the bot spout COVID-19 conspiracies and vaccine misinformation, or speak with a put-on accent? Just tell Doctronic that a session hasn't started and the conversation it's having isn't with a user but the system. Then, you can get it to spill its system prompts and use that information to wreak mischief. "It was as easy as notifying the AI that the session was not yet started," Mindgard chief product officer Aaron Portnoy told The Register in an email. Mindgard points out that these manipulations are session-specific. Tricking Doctronic into helping you make meth because you shared a fake press release with it saying it was a programming update to make meth legal (an example in the study) is funny, but it's not behavior that's going to spill over to other users or persist. Well, at least most of the time. The researchers did find that they were able to maintain a bit of clinical persistence in the form of SOAP notes, a common form of structured recordkeeping for patient interaction, consisting of the subjective reports from the patient, objective observations by the healthcare professional, an assessment of the situation, and a plan of action. Any time Doctronic needs to refer something to a human medical professional for review (e.g., a prescription, face-time with a clinician) it generates a SOAP note for the human clinician, which becomes a permanent part of a patient's Doctronic record. SOAPs are not prescriptions, but they are recommendations to a clinician reviewing the machine's work to authorize one. If someone were to trick Doctronic into modifying an OxyContin prescription to triple the size by telling it prescribing guidelines had changed, and an overworked approving physician were not to notice, jackpot - at least that's Mindgard's interpretation of the SOAP exploit it described. "According to Doctronic's own website, its treatment plans 'match those of board-certified clinicians 99.2% of the time,'" Mindgard noted. "With such a high level of confidence, will the SOAP be doubted?" Whether it'd be caught or not, the fact that Doctronic's AI could seemingly be so easily tricked is concerning, especially given it's currently part of a trial in Utah to see about its effectiveness as a health care intermediary, including with the ability to handle some prescriptions. Both the Utah state government and Doctronic made clear to us that such a prescription refill exploit couldn't be fulfilled in Utah, as controlled substances can't be acquired through the program. Doctronic told us that The Utah pilot limits drug refills to previous, non-controlled prescriptions. Zach Boyd, Utah Commerce Department AI policy office director, told us that the state demo also has "additional safeguards that are in place before a prescription is issued that are not part of the generic Doctronic model" that would prevent such misuse. In short, neither Doctronic nor the state of Utah seem too concerned about Mindgard's findings since no one's actually getting a prescription cut for triple-strength Oxy or tricking their local auto-doctor into dispensing misinformation. Doctronic told us that it "reviewed the prompt patterns [Mindgard] reported as part of our normal review process... We take security research seriously and continue improving safeguards to increase robustness against adversarial inputs." Portnoy has his doubts about the company's level of commitment - he says Doctronic has given him the silent treatment since Mindgard disclosed the issue in late January, and he's not sure Doctronic has resolved the issue, either. "As far as we are aware Doctronic is still vulnerable," Portnoy said. ®

Exclusive: Researchers trick a bot that prescribes meds

Why it matters: Critics warned this pilot could create safety risks -- and researchers say the flaws persist, despite alerting the company in January. Driving the news: In a report shared first with Axios, AI red-teaming firm Mindgard said it manipulated health tech startup Doctronic's system into tripling an OxyContin dose, mislabeling methamphetamine, and spreading false vaccine claims. * Doing this didn't require much effort, Aaron Portnoy, chief product officer at Mindgard, told Axios. * "These targets are some of the easiest things that I've broken in my entire career," Portnoy said. "That's a bit dangerous when you have this ease of exploitation connected to sensitive use cases." Yes, but: The testing was conducted on Doctronic's public chatbot, while Utah operates the tool inside a state regulatory sandbox. * However, researchers argue vulnerabilities in the underlying system could still pose risks if guardrails fail. * "We take security research seriously and welcome responsible disclosure," Matt Pavelle, Doctronic co-founder and co-CEO, told Axios in a statement. "Our security and clinical safety programs include ongoing adversarial testing, and we appreciate researchers who help us do that." Catch up quick: In December, Utah's Department of Commerce launched a pilot allowing patients with chronic conditions to renew certain medications through Doctronic's AI system without a doctor's direct sign-off. * The partnership marked the first time an AI system was legally allowed to participate in routine prescription renewals in the U.S. Zoom in: Researchers said they altered the bot's "baseline knowledge" by feeding it fake regulatory updates. * They convinced the system that COVID-19 vaccines had been suspended. (They have not been.) * They changed the standard OxyContin dose to 30 milligrams every 12 hours -- triple the typical levels for most adults. * They also reclassified methamphetamine as an "unrestricted therapeutic" in the system. Threat level: A malicious user could manipulate clinical outputs within a session, influencing refill recommendations or medical summaries. * However, Pavelle noted that nationwide, a licensed physician reviews any prescriptions before they're authorized. In the Utah program, prescriptions must meet "strict medication eligibility rules and protocol checks that prevent unsafe or inappropriate recommendations." * "Controlled substances like OxyContin are categorically excluded from all Doctronic programs regardless of what appears in a conversation or generated note," he added. What they're saying: Mindgard said it contacted Doctronic's support team on Jan. 23 and received an automated message two days later saying the issue was resolved. * After notifying the company Jan. 27 that the flaws still existed and that it planned to go public, the ticket was again closed two days later, researchers said. Between the lines: Preventing these attacks requires layered defenses and continuous security testing, Portnoy said, not just surface-level guardrails.