Utah's AI Prescription Pilot Sparks Medical Debate as Doctronic Bypasses Traditional Licensing

Utah's AI prescription pilot alarms its medical board

Utah has become the first US state to let an AI chatbot, Doctronic, renew prescriptions without a doctor, via a regulatory sandbox that waives licensing laws. The state's medical licensing board, blindsided by the January launch, called in April for the pilot to be halted over safety risks, but the state refused. The case exposes a federal-state regulatory vacuum around AI in medicine. Utah has quietly become the first US state to let an AI chatbot renew prescriptions without a doctor, according to the Associated Press. The programme, run by a company called Doctronic, launched in January and has set off a fierce medical debate. Residents can skip the doctor's office and refill prescriptions online through the chatbot. It asks about their medication and history, checks a national pharmacy database, and either renews the script or escalates to a human doctor. The launch was possible only through a "regulatory sandbox" that lets Utah officials waive laws for promising AI. State and federal rules otherwise restrict prescribing to licensed medical professionals. "We have crossed a threshold in terms of giving something that is not human a medical license, whether or not we want to call it that," the University of Pennsylvania's Dr Eric Bressman told the AP. He and others say they are not opposed to AI prescribing, but want it held to standards as rigorous as those for human doctors. The board that got left out Utah's medical licensing board says it only learned of the programme when the January launch made the news. In an April letter, 11 members called for the pilot to be halted, citing the risks of auto-renewing drugs with side effects or interactions. "We were essentially told: 'Yes this is going on. And no, you don't have a say in it'," said Dr Alan Smith, a family physician who chairs the board but spoke for himself. The state declined to suspend it, noting human doctors still review every refill in this first phase. The programme is currently overseen by a five-member board of AI specialists, none of them doctors. Doctronic expects to move to fully automated refills soon. Smith warns the risks are real, pointing out that Doctronic's roughly 190 refillable medications include blood thinners, which turn dangerous if a patient develops internal bleeding. The American Medical Association has echoed the concern that "prescription renewals aren't routine checkboxes". A regulatory vacuum by design The case exposes a jurisdictional tangle, since medical technology is regulated federally while medical professionals are overseen by states. Doctronic frames its AI as part of state-regulated medical practice, though some experts argue it has crossed into FDA territory. The company would not say whether it has sought FDA permission. The agency told the AP it has authorised no AI chatbots but wants to encourage innovation, a hands-off posture that fits a broader loosening of oversight on AI health tools. Critics see history rhyming, with Bressman comparing the moment to the haphazard medicine of the early 20th century, before boards and benchmarks existed. The template for licensing AI medical services in other states comes from the Cicero Institute, a pro-AI think tank founded by Palantir co-founder Joe Lonsdale. The stakes are not abstract, as safety researchers have warned that medical chatbots can sound authoritative while dispensing dangerous advice. Others caution that removing humans from care can undermine the very outcomes it promises. Rivals are scrambling to map those failure modes too. Meta went as far as posing as teenagers to test how competing chatbots handle sensitive topics. Doctronic plans peer-reviewed studies later this year, though its only published paper so far was written by its own scientists and not independently reviewed. As one Utah law professor put it, companies risk letting the technology race beyond the evidence, and betraying public trust in the process.

Can AI fill prescriptions? Here's what doctor's think of Utah's refill program

The program allows Utah residents to skip the doctor's office and get their prescriptions refilled online by an AI chatbot called Doctronic. It's a seemingly simple step toward making healthcare more convenient for patients and prescribers. But it's also a precedent-shattering milestone that has set off alarm bells for doctors, lawyers and public health experts. The pilot program has laid bare a host of questions about the role of AI in medicine, including how it should be regulated, whether doctors should be able to veto it, and what kind of safety measures are needed to protect patients. At the center of the debate: state and federal laws limit prescribing to licensed medical professionals. Proponents say those laws, which have underwritten American medicine for over 100 years, should be updated to include AI chatbots and other new technologies.

Is AI Ready to Take Over Your Prescriptions? Doctors Are Wary of Utah's Automated Refill Program

This illustration provided by Doctronic depicts the process for refilling prescriptions online with an AI chatbot on a smartphone. (Doctronic via AP) WASHINGTON (AP) -- A prescription refill program that quietly launched in Utah earlier this year has kicked off a big medical debate: Is artificial intelligence ready to take over tasks that, until now, could only be performed by doctors? The program allows Utah residents to skip the doctor's office and get their prescriptions refilled online by an AI chatbot called Doctronic. It's a seemingly simple step toward making healthcare more convenient for patients and prescribers. But it's also a precedent-shattering milestone that has set off alarm bells for doctors, lawyers and public health experts. The pilot program has laid bare a host of questions about the role of AI in medicine, including how it should be regulated, whether doctors should be able to veto it, and what kind of safety measures are needed to protect patients. At the center of the debate: state and federal laws limit prescribing to licensed medical professionals. Proponents say those laws, which have underwritten American medicine for over 100 years, should be updated to include AI chatbots and other new technologies. "We have crossed a threshold in terms of giving something that is not human a medical license, whether or not we want to call it that," said Dr. Eric Bressman of the University of Pennsylvania. AI cannot practice medicine under current laws Bressman and other experts say they aren't opposed to AI prescribing. But they say it should have to meet rigorous standards akin to human doctors, who undergo years of testing and training before being licensed to practice medicine. In Utah, Doctronic was able to launch thanks to a "regulatory sandbox" that allows state officials to waive laws for AI companies offering promising technology. The refill program is currently overseen by a five-member board of AI specialists, none of whom are doctors, who say they have implemented numerous safeguards. During the program's initial phase, for example, human doctors review all Doctronic refill orders. The company expects to soon transition to fully automated refills. The head of the state's medical licensing board says he and his colleagues learned of the program when its January launch was reported in the news. In a March letter to the state, 11 board members called for the program to be halted, citing the risks of automatically renewing medicines that can have side effects or drug interactions. "We were essentially told: 'Yes this is going on. And no, you don't have a say in it,'" said Dr. Alan Smith, a family physician who heads the board but said he was speaking only for himself. Complicating the picture is the fact that medical technology is traditionally regulated at the federal level, while medical professionals are overseen by states. Doctronic executives consider their AI part of the state-regulated practice of medicine. But the federal Food and Drug Administration is supposed to oversee AI that directly impacts medical care or decision making, a line that some experts believe Doctronic has crossed. Some states are clearing the way for AI in healthcare In an interview, Doctronic's executives wouldn't say whether they have sought permission from the FDA. "Our goal here is really just to meet patients where they need healthcare," said Dr. Adam Oskowitz, who co-founded the company with a tech industry entrepreneur. "We try not to get too deep into the weeds on the regulatory side." In Utah, residents can visit a Doctronic website built for the refill program. After confirming their identity, the AI chatbot asks users about their prescriptions and medical history, verifying that they have a valid prescription by tapping into a national pharmacy database. If there are no issues, the AI can renew the prescription and send it to a local pharmacy. If the request requires more attention, the chatbot transfers the patient to a doctor who works for Doctronic's telehealth service. Oskowitz envisions a future where many routine medical tasks, including ordering tests and analyzing results, can be offloaded to Doctronic, allowing doctors to manage thousands more patients than they can today. Other states are also waiving rules for AI, including Texas and Wyoming. Meanwhile, lawmakers in Iowa, Idaho and elsewhere have introduced legislation to formally license AI medical services. Many of the bills are based on a template from the nonprofit Cicero Institute, a pro-AI think tank founded by Joe Lonsdale, co-founder of the artificial intelligence software company Palantir. Pushback against medical AI mainly stems from the economic fears of doctors and other health workers, says Cicero's director for health policy. "Whoever goes first is going to take the slings and arrows because there's economic interests, concerns about the workforce and what that's going to mean for jobs," said Cicero's Adam Meier. Doctors see potential risks to AI prescription refills Smith, the medical board chair, says the risks to patients are real. He points out that Doctronic's list of 190 refillable medications includes blood thinners, which can become dangerous if patients develop stomach ulcers or other conditions that cause internal bleeding. "Many times when I see people after six months I find that their medical history or situation has changed," Smith said. "Just because something was prescribed before does not mean it's appropriate now." The American Medical Association has voiced similar concerns, warning that "prescription renewals aren't routine checkboxes." Zach Boyd, who heads Utah's AI office, said Doctronic has thus far been overly cautious, often elevating uncontroversial decisions to doctors. In response to safety concerns, several medications have been removed from the list eligible for refills, including a drug for irregular heartbeats. Utah has released some initial data on the program and Doctronic plans to publish peer-reviewed studies later this year. Currently the only publication about its technology is a paper written by company scientists that was not independently reviewed. The study looked at whether Doctronic could correctly diagnose medical conditions based on records from 500 telehealth consultations. In the study, Doctronic's diagnoses matched that of human doctors 80% of the time. The FDA is taking a hands-off approach Bressman says Utah should have demanded data on prescription refills up front, not after Doctronic was up and running. "Mostly they're accepting the company's word on good faith that they're up to the task," he said. The current approach to AI mirrors the haphazard medical standards of the early 20th century, Bressman says, before medical schools, medical boards and other authorities agreed on national benchmarks for training and licensing. National guidelines on medical technology would typically come from the FDA, but the agency has indicated it plans to take a hand-off approach, at least under the current administration. An FDA spokesperson said the agency has not authorized any AI chatbots but "is committed to encouraging medical innovation and helping bring promising new technologies to patients, while keeping safety at the center of every decision." For now, Doctronic and other companies are likely to expand across states with different regulatory approaches. "Companies may benefit in the short term by expanding their business models and kind of having the technology go beyond the evidence," says Daniel Aaron of University of Utah's law school. "But in the long-term, I think they risk compromising public trust and fueling backlash." ___ The Associated Press Health and Science Department receives support from the Howard Hughes Medical Institute's Department of Science Education and the Robert Wood Johnson Foundation. The AP is solely responsible for all content.

States Move to License AI Doctors as the FDA Steps Back | PYMNTS.com

Six months later, the pilot has become a national flashpoint. Physicians, patient safety advocates and regulators are raising questions the technology cannot answer on its own. Who is responsible when AI misses a contraindication? When should the software refuse a request? And where does a state-regulated clinical tool end and a federally regulated medical device begin? The safety concerns are specific. Dr. Alan Smith, a family physician who chairs Utah's medical board, told the Associated Press that Doctronic's list of 190 refillable medications includes blood thinners, which can become dangerous if patients develop conditions that cause internal bleeding. "Many times when I see people after six months I find that their medical history or situation has changed," Smith said. "Just because something was prescribed before does not mean it's appropriate now." The American Medical Association has echoed the concern, warning that "prescription renewals aren't routine checkboxes." Within weeks of its launch, security researchers were able to push the Doctronic system to triple a patient's opioid dosage and generate vaccine misinformation, The Verge reported in April. AI's Clinical Role Collides With a Century of Licensing Law Doctronic executives would not say whether they have sought permission from the U.S. Food and Drug Administration (FDA) for the system, the AP reported. "Our goal here is really just to meet patients where they need healthcare," Co-Founder Dr. Adam Oskowitz told the AP. "We try not to get too deep into the weeds on the regulatory side." The FDA, which typically sets national guidelines for medical technology, plans to take a hands-off approach to regulating AI prescription tools for now, according to the AP. An FDA spokesperson said the agency has not authorized any AI chatbots but is "committed to encouraging medical innovation and helping bring promising new technologies to patients, while keeping safety at the center of every decision." Meanwhile, lawmakers in Iowa and Idaho have advanced bills, including Idaho House Bill 945 and Iowa HSB 766, to create licensing systems for autonomous AI healthcare providers. These efforts require audits and aim to regulate systems authorized to diagnose or prescribe without human supervision. Dr. Eric Bressman of the University of Pennsylvania framed the structural concern: "We have crossed a threshold in terms of giving something that is not human a medical license,"whether or not we want to call it that," he told the AP. A Second Utah Pilot Narrows the Scope. The Questions Remain. Utah approved a second AI prescription pilot in April, this time through Y Combinator-backed Legion Health, focused on psychiatric medication renewals in a state where up to 500,000 residents lack access to adequate behavioral healthcare, according to the Utah Office of AI Policy, PYMNTS reported. Legion's guardrails are tighter than Doctronic's. The AI cannot issue new prescriptions, adjust doses or handle controlled substances. Patients must be stable, on an existing treatment plan with a licensed psychiatrist and must not have had a psychiatric hospitalization in the past year. The demand side is not in question. More than 40 million people worldwide use ChatGPT daily for health-related queries, with about 70% of those interactions happening outside clinic hours, PYMNTS reported. Daniel Aaron, a law professor at the University of Utah, captured the risk of moving faster than the evidence. "Companies may benefit in the short term by expanding their business models and kind of having the technology go beyond the evidence," Aaron told the AP. "But in the long term, I think they risk compromising public trust and fueling backlash." For all PYMNTS AI and digital transformation coverage, subscribe to the daily AI and Digital Transformation Newsletters.