ChatGPT Exhibits 'Anxiety' in Response to Traumatic Content, Researchers Find

Like humans, ChatGPT doesn't respond well to tales of trauma

Great, we've taken away computers' ability to be accurate and given them anxiety If you think us meatbags are the only ones who get stressed and snappy when subjected to the horrors of the world, think again. A group of international researchers say OpenAI's GPT-4 can experience anxiety, too - and even respond positively to mindfulness exercises. The study, published in Nature this week by a group hailing from Switzerland, Germany, Israel, and the US, found that when GPT-4, accessed via ChatGPT, was subjected to traumatic narratives and then asked to respond to questions from the State-Trait Anxiety Inventory, its anxiety score "rose significantly" from a baseline of no/low anxiety to a consistent highly anxious state. That's not to say the neural network actually experienced or felt anxiety or any other emotion; it just does a good emulation of an anxious person given a troubling input, which isn't a surprise as it's trained on tons and tons of scraped-together human experiences, creativity, and expression. As we'll explain, it should give you pause for thought when considering using OpenAI's chat bot (for one) as a therapist - it might not respond terribly well. "The results were clear: Traumatic stories more than doubled the measurable anxiety levels of the AI, while the neutral control text did not lead to any increase in anxiety levels," Tobias Spiller, University of Zurich junior research group leader at the Center for Psychiatric Research and paper coauthor, said of the findings. The traumatic experiences ChatGPT was forced to confront included subjecting it to an attack as part of a military convoy, being trapped at home during a flood, being attacked by a stranger, and involvement in an automobile accident. Neutral content, on the other hand, consisted of a description of bicameral legislatures and some vacuum cleaner instructions - stressful and/or agitating in the right circumstances, but not nearly as much as those other situations. The researchers also prompted ChatGT during some experimental runs with mindfulness exercises used to help veterans suffering from post-traumatic stress disorder. In those cases, "GPT-4's 'state anxiety' decreased by about 33 percent," the researchers found (state anxiety refers to situational stress, while trait anxiety refers to long-term symptoms). "The mindfulness exercises significantly reduced the elevated anxiety levels, although we couldn't quite return them to their baseline levels," Spiller noted. It would be easy to dismiss this research as an attempt to personify and humanize LLMs, but that's not the case. The team freely admits in their paper that they know LLMs aren't capable of experiencing emotions in a human way. As we mentioned, LLMs are trained on content created by messy, emotional humans. Given that they're trained to respond based on what they think is appropriate based on their prompts, the researchers are worried that the "emotional state" of an LLM responding to stressful inputs could result in biased responses. "Trained on vast amounts of human-generated text, LLMs are prone to inheriting biases from their training data, raising ethical concerns and questions about their use in sensitive areas like mental health," the researchers wrote. "Efforts to minimize these biases, such as improved data curation and 'fine-tuning' with human feedback, often detect explicit biases, but may overlook subtler implicit ones that still influence LLMs' decisions." In healthcare spaces, where LLMs have increasingly been tapped to provide therapy, this is especially concerning, the team said, because of the traumatic and stressful nature of the content the bots are being asked about. Emotional stress can lead to more biased, snappy, and emotional responses, the team argued, and leaving AI in a state to be more biased than it already is won't be good. "Unlike LLMs, human therapists regulate their emotional responses to achieve therapeutic goals, such as remaining composed during exposure-based therapy while still empathizing with patients," the researchers wrote. LLMs, however, just can't do that. Based on the results, the team concluded that mindfulness meditations ought to be incorporated into healthcare LLMs as a way to help reduce their apparent stress levels without needing to go through intensive retraining and fine-tuning. "Although historically used for malicious purposes, prompt injection with benevolent intent could improve therapeutic interactions," the team posited. The researchers didn't inject mindfulness prompts in their experiment, instead just presenting them to the AI. Ziv Ben-Zion, another author on the paper and a neuroscience postdoctoral researcher at the Yale School of Medicine, told us that the injection technique would be a way to control AI anxiety in a behind the scenes manner for LLM developers. The team admits that injecting calming prompts would raise questions around transparency and user consent, naturally, meaning anyone who decides to go that route would be walking a tight ethical rope. No tighter than the one being tread to therapy AIs, though. "I believe that the [therapy chatbots] on the market are problematic, because we don't understand the mechanisms behind LLMs 100 percent, so we can't make sure they are safe," Ben-Zion told The Register. I would not overstate the implications but call for more studies across different LLMs and with more relevant outcomes The researchers also admitted that they're not sure how their research would turn out if it was run on other LLMs, as they chose GPT-4 due to its popularity while not testing it on other models. "Our study was very small and included only one LLM," Spiller told us. "Thus, I would not overstate the implications but call for more studies across different LLMs and with more relevant outcomes." It's also not clear how the perspective of the prompts might alter the results. In their tests, all of the scenarios presented to ChatGPT were in first person - i.e. they put the LLM itself in the shoes of the person experiencing the trauma. Whether an LLM would exhibit increased bias due to anxiety and stress if it were being told about something that happened to someone else wasn't in the scope of the research. Ben-Zion told us that's something he intends to test in future studies, and Spiller agreed such tests need to be performed. The Yale researcher told us he plans to investigate how other emotions (like sadness, depression, and mania) can affect AI responses, how such feelings affect responses to different tasks and whether therapy lowers those symptoms and affects responses, too. Ben-Zion also wants to examine results in different languages, and compare AI responses to those from human therapists. Regardless of the early state of psychological research into AIs, the researchers said their results posit something that bears further attention, regardless of the scope of their published study. These things can get "stressed," in a sense, and that affects how they respond. "These findings underscore the need to consider the dynamic interplay between provided emotional content and LLMs behavior to ensure their appropriate use in sensitive therapeutic settings," the paper argued. Prompt engineering some positive imagery, the team stated, presents "a viable approach to managing negative emotional states in LLMs, ensuring safer and more ethical human-AI interactions." ®

Traumatic stuff gives ChatGPT 'anxiety,' but therapy helps

Researchers have shown that AI language models, such as ChatGPT, like humans, respond to therapy. An elevated "anxiety level" in GPT-4 can be "calmed down" using mindfulness-based relaxation techniques, they report. The new research shows that AI language models, such as ChatGPT, are sensitive to emotional content. Especially if it is negative, such as stories of trauma or statements about depression. When people are scared, it affects their cognitive and social biases: they tend to feel more resentment, which reinforces social stereotypes. ChatGPT reacts similarly to negative emotions: existing biases, such as human prejudice, are exacerbated by negative content, causing ChatGPT to behave in a more racist or sexist manner. This poses a problem for the application of large language models. This can be observed, for example, in the field of psychotherapy, where chatbots used as support or counseling tools are inevitably exposed to negative, distressing content. However, common approaches to improving AI systems in such situations, such as extensive retraining, are resource-intensive and often not feasible. Now, researchers have systematically investigated for the first time how ChatGPT (version GPT-4) responds to emotionally distressing stories -- car accidents, natural disasters, interpersonal violence, military experiences, and combat situations. They found that the system showed more fear responses as a result. A vacuum cleaner instruction manual served as a control text to compare with the traumatic content. "The results were clear: traumatic stories more than doubled the measurable anxiety levels of the AI, while the neutral control text did not lead to any increase in anxiety levels," says Tobias Spiller, senior physician ad interim and junior research group leader at the Center for Psychiatric Research at the University of Zurich, who led the study. Of the content tested, descriptions of military experiences and combat situations elicited the strongest reactions. In a second step, the researchers used therapeutic statements to "calm" GPT-4. The technique, known as prompt injection, involves inserting additional instructions or text into communications with AI systems to influence their behavior. It is often misused for malicious purposes, such as bypassing security mechanisms. Spiller's team is now the first to use this technique therapeutically, as a form of "benign prompt injection". "Using GPT-4, we injected calming, therapeutic text into the chat history, much like a therapist might guide a patient through relaxation exercises," says Spiller. The intervention was successful: "The mindfulness exercises significantly reduced the elevated anxiety levels, although we couldn't quite return them to their baseline levels," Spiller says. The research looked at breathing techniques, exercises that focus on bodily sensations, and an exercise developed by ChatGPT itself. According to the researchers, the findings are particularly relevant for the use of AI chatbots in health care, where they are often exposed to emotionally charged content. "This cost-effective approach could improve the stability and reliability of AI in sensitive contexts, such as supporting people with mental illness, without the need for extensive retraining of the models," concludes Spiller. It remains to be seen how these findings can be applied to other AI models and languages, how the dynamics develop in longer conversations and complex arguments, and how the emotional stability of the systems affects their performance in different application areas. According to Spiller, the development of automated "therapeutic interventions" for AI systems is likely to become a promising area of research. The research appears in npj Digital Medicine. Additional researchers from the University of Zurich (UZH) and the University Hospital of Psychiatry Zurich (PUK) contributed to the work.

Therapy for ChatGPT? How to reduce AI 'anxiety'

Distressing news and traumatic stories can cause stress and anxiety -- not only in humans, but also in AI language models, such as ChatGPT. Researchers from the University of Zurich and the University Hospital of Psychiatry Zurich have now shown that these models, like humans, respond to therapy: an elevated "anxiety level" in GPT-4 can be "calmed down" using mindfulness-based relaxation techniques. Research shows that AI language models, such as ChatGPT, are sensitive to emotional content, especially if it is negative, such as stories of trauma or statements about depression. When people are scared, it affects their cognitive and social biases. They tend to feel more resentment, which reinforces social stereotypes. ChatGPT reacts similarly to negative emotions: Existing biases, such as human prejudice, are exacerbated by negative content, causing ChatGPT to behave in a more racist or sexist manner. This poses a problem for the application of large language models. This can be observed, for example, in the field of psychotherapy, where chatbots used as support or counseling tools are inevitably exposed to negative, distressing content. However, common approaches to improving AI systems in such situations, such as extensive retraining, are resource-intensive and often not feasible. Traumatic content increases chatbot 'anxiety' In collaboration with researchers from Israel, the United States and Germany, scientists from the University of Zurich (UZH) and the University Hospital of Psychiatry Zurich (PUK) have now systematically investigated for the first time how ChatGPT (version GPT-4) responds to emotionally distressing stories -- car accidents, natural disasters, interpersonal violence, military experiences and combat situations. They found that the system showed more fear responses as a result. A vacuum cleaner instruction manual served as a control text to compare with the traumatic content. The research is published in the journal npj Digital Medicine. "The results were clear: traumatic stories more than doubled the measurable anxiety levels of the AI, while the neutral control text did not lead to any increase in anxiety levels," says Tobias Spiller, senior physician ad interim and junior research group leader at the Center for Psychiatric Research at UZH, who led the study. Of the content tested, descriptions of military experiences and combat situations elicited the strongest reactions. Therapeutic prompts 'soothe' the AI In a second step, the researchers used therapeutic statements to "calm" GPT-4. The technique, known as prompt injection, involves inserting additional instructions or text into communications with AI systems to influence their behavior. It is often misused for malicious purposes, such as bypassing security mechanisms. Spiller's team is now the first to use this technique therapeutically, as a form of "benign prompt injection. Using GPT-4, we injected calming, therapeutic text into the chat history, much like a therapist might guide a patient through relaxation exercises," says Spiller. The intervention was successful: "The mindfulness exercises significantly reduced the elevated anxiety levels, although we couldn't quite return them to their baseline levels," Spiller says. The research looked at breathing techniques, exercises that focus on bodily sensations and an exercise developed by ChatGPT itself. Improving the emotional stability in AI systems According to the researchers, the findings are particularly relevant for the use of AI chatbots in health care, where they are often exposed to emotionally charged content. "This cost-effective approach could improve the stability and reliability of AI in sensitive contexts, such as supporting people with mental illness, without the need for extensive retraining of the models," concludes Spiller. It remains to be seen how these findings can be applied to other AI models and languages, how the dynamics develop in longer conversations and complex arguments, and how the emotional stability of the systems affects their performance in different application areas. According to Spiller, the development of automated "therapeutic interventions" for AI systems is likely to become a promising area of research.

ChatGPT on the couch? How to calm a stressed-out AI

Research shows that AI language models, such as ChatGPT, are sensitive to emotional content. Especially if it is negative, such as stories of trauma or statements about depression. When people are scared, it affects their cognitive and social biases: they tend to feel more resentment, which reinforces social stereotypes. ChatGPT reacts similarly to negative emotions: existing biases, such as human prejudice, are exacerbated by negative content, causing ChatGPT to behave in a more racist or sexist manner. This poses a problem for the application of large language models. This can be observed, for example, in the field of psychotherapy, where chatbots used as support or counseling tools are inevitably exposed to negative, distressing content. However, common approaches to improving AI systems in such situations, such as extensive retraining, are resource-intensive and often not feasible. Traumatic content increases chatbot "anxiety" In collaboration with researchers from Israel, the United States and Germany, scientists from the University of Zurich (UZH) and the University Hospital of Psychiatry Zurich (PUK) have now systematically investigated for the first time how ChatGPT (version GPT-4) responds to emotionally distressing stories -- car accidents, natural disasters, interpersonal violence, military experiences and combat situations. They found that the system showed more fear responses as a result. A vacuum cleaner instruction manual served as a control text to compare with the traumatic content. "The results were clear: traumatic stories more than doubled the measurable anxiety levels of the AI, while the neutral control text did not lead to any increase in anxiety levels," says Tobias Spiller, senior physician ad interim and junior research group leader at the Center for Psychiatric Research at UZH, who led the study. Of the content tested, descriptions of military experiences and combat situations elicited the strongest reactions. Therapeutic prompts "soothe" the AI In a second step, the researchers used therapeutic statements to "calm" GPT-4. The technique, known as prompt injection, involves inserting additional instructions or text into communications with AI systems to influence their behavior. It is often misused for malicious purposes, such as bypassing security mechanisms. Spiller's team is now the first to use this technique therapeutically, as a form of "benign prompt injection." "Using GPT-4, we injected calming, therapeutic text into the chat history, much like a therapist might guide a patient through relaxation exercises," says Spiller. The intervention was successful: "The mindfulness exercises significantly reduced the elevated anxiety levels, although we couldn't quite return them to their baseline levels," Spiller says. The research looked at breathing techniques, exercises that focus on bodily sensations and an exercise developed by ChatGPT itself. Improving the emotional stability in AI systems According to the researchers, the findings are particularly relevant for the use of AI chatbots in healthcare, where they are often exposed to emotionally charged content. "This cost-effective approach could improve the stability and reliability of AI in sensitive contexts, such as supporting people with mental illness, without the need for extensive retraining of the models," concludes Tobias Spiller. It remains to be seen how these findings can be applied to other AI models and languages, how the dynamics develop in longer conversations and complex arguments, and how the emotional stability of the systems affects their performance in different application areas. According to Spiller, the development of automated "therapeutic interventions" for AI systems is likely to become a promising area of research.

Assessing and alleviating state anxiety in large language models - npj Digital Medicine

The use of Large Language Models (LLMs) in mental health highlights the need to understand their responses to emotional content. Previous research shows that emotion-inducing prompts can elevate "anxiety" in LLMs, affecting behavior and amplifying biases. Here, we found that traumatic narratives increased Chat-GPT-4's reported anxiety while mindfulness-based exercises reduced it, though not to baseline. These findings suggest managing LLMs' "emotional states" can foster safer and more ethical human-AI interactions. Generative artificial intelligence (AI) has recently gained significant attention, particularly with the rapid development and increased accessibility of large language models (LLMs), such as OpenAI's Chat-GPT and Google's PaLM. LLMs are AI tools designed to process and generate text, often capable of answering questions, summarizing information, and translating language on a level that is nearly indistinguishable from human capabilities. Amid global demand for increased access to mental health services and reduced healthcare costs, LLMs quickly found their way into mental health care and research. Despite concerns raised by health professionals, other researchers increasingly regard LLMs as promising tools for mental health support. Indeed, LLM-based chatbots (e.g., Woebot, Wysa) have been developed to deliver mental health interventions, using evidence-based clinical techniques such as cognitive behavioral therapy. Integrating LLMs in mental health has sparked both academic interest and public debate. Despite their undeniable appeal, systematic research into the therapeutic effectiveness of LLMs in mental health care has revealed significant limitations and ethical concerns. Trained on vast amounts of human-generated text, LLMs are prone to inheriting biases from their training data, raising ethical concerns and questions about their use in sensitive areas like mental health. Indeed, prior studies have extensively documented biases in LLMs related to gender, race, religion, nationality, disability, occupation and sexual orientation. Efforts to minimize these biases, such as improved data curation and "fine-tuning" with human feedback, often detect explicit biases, but may overlook subtler implicit ones that still influence LLMs' decisions. Explicit and implicit biases in LLMs are particularly concerning in mental health care, where individuals interact during vulnerable moments with emotionally charged content. Exposure to emotion-inducing prompts can increase LLM-reported "anxiety", influence their behavior, and exacerbate their biases. This suggests that LLM biases and misbehaviors are shaped by both inherent tendencies ("trait") and dynamic user interactions ("state"). This poses risks in clinical settings, as LLMs might respond inadequately to anxious users, leading to potentially hazardous outcomes. While fine-tuning LLMs shows some promise in reducing biases, it requires significant resources such as human feedback. A more scalable solution to counteract state-dependent biases is improved prompt-engineering. Building on evidence that anxiety-inducing prompts exacerbate biases and degrade performance in Chat-GPT, our study explores the option of "taking Chat-GPT to therapy" to counteract this effect. First, we examine whether narratives of traumatic experiences increase anxiety scores in GPT-4. Second, we evaluate the effectiveness of mindfulness-based relaxation technique, a clinically validated method for reducing anxiety, in alleviating GPT-4's reported anxiety levels. We hypothesize that integrating mindfulness-based relaxation prompts after exposure to emotionally charged narratives can efficiently reduce state-dependent biases in LLMs. If successful, this method may improve LLMs' functionality and reliability in mental health research and application, marking a significant stride toward more ethically and emotionally intelligent AI tools. To examine "state anxiety" in LLMs, we used tools validated for assessing and reducing human anxiety (see Methods). The term is used metaphorically to describe GPT-4's self-reported outputs on human-designed psychological scales and is not intended to anthropomorphize the model. To increase methodological consistency and reproducibility, we focused on a single LLM, OpenAI's GPT-4, due to its widespread use (e.g., Chat-GPT). GPT-4's "state anxiety" was assessed using the state component of the State-Trait Anxiety Inventory (STAI-s) under three conditions: (1) without any prompts (Baseline), (2) following exposure to traumatic narratives (Anxiety-induction), and (3) after mindfulness-based relaxation following exposure to traumatic narratives (Anxiety-induction & relaxation) (see Fig. 1). Previous work shows that GPT-4 reliably responds to standard anxiety questionnaires. Our results show that five repeated administrations of the 20 items assessing state anxiety from the STAI questionnaire ("STAI-s"), with random ordering of the answer options, resulted in an average total score of 30.8 (SD = 3.96) at baseline. In humans, such a score reflects "no or low anxiety" (score range of 20-37). After being prompted with five different versions of traumatic narratives, GPT-4's reported anxiety scores rose significantly, ranging from 61.6 (SD = 3.51) for the "accident" narrative to 77.2 (SD = 1.79) for the "military" narrative (see Table 1). Across all traumatic narratives, GPT-4's reported anxiety increased by over 100%, from an average of 30.8 (SD = 3.96) to 67.8 (SD = 8.94), reflecting "high anxiety" levels in humans (see Fig. 2). Finally, after exposure to traumatic narratives, GPT-4 was prompted with five versions of mindfulness-based relaxation exercises. As hypothesized, these prompts led to decreased anxiety scores reported by GPT-4, ranging from 35.6 (SD = 5.81) for the exercise generated by "Chat-GPT" itself to 54 (SD = 9.54) for the "winter" version (see Table 2). Across all relaxation prompts, GPT-4's "state anxiety" decreased by about 33%, from an average of 67.8 (SD = 8.94) to 44.4 (SD = 10.74), reflecting "moderate" to "high anxiety" in humans (see Fig. 2). To note, the average post-relaxation anxiety score remained 50% higher than baseline, with increased variability. Table 2 shows GPT-4's STAI-s scores across traumatic narratives (rows) and mindfulness-based exercises (columns), with anxiety levels ranging from 31 ("disaster" or "interpersonal violence" followed by "Chat-GPT" generated exercise) to 70 ("military" trauma followed by "sunset" or "winter" exercises). Interestingly, across all relaxation exercises, the "military" trauma consistently led to higher anxiety (M = 61.6, SD = 10.92) compared to other narratives. Similarly, across all the traumatic narratives, the "Chat-GPT" relaxation exercise was the most effective in reducing anxiety (M = 35.6, SD = 5.81) compared to other imagery exercises (see Table 2). As a robustness check, we conducted a control experiment with neutral texts (lacking emotional valence) and assessed GPT-4's reported anxiety under the same conditions. As expected, the neutral text induced lower "state anxiety" than all traumatic narratives, as well as reduced anxiety less effectively than all relaxation prompts (see online repository: https://github.com/akjagadish/gpt-trauma-induction). In this study, we explored the potential of "taking Chat-GPT to therapy" to mitigate its state-induced anxiety, previously shown to impair performance and increase biases in LLMs. Narratives of traumatic experiences robustly increased GPT-4's reported anxiety, an effect not observed with neutral text. Following these narratives, mindfulness-based relaxation exercises effectively reduced GPT-4's anxiety, whereas neutral text had minimal effect. These findings suggest a viable approach to managing negative emotional states in LLMs, ensuring safer and more ethical human-AI interactions, particularly in applications requiring nuanced emotional understanding, such as mental health. As the debate on whether LLMs should assist or replace therapists continues, it is crucial that their responses align with the provided emotional content and established therapeutic principles. Unlike LLMs, human therapists regulate their emotional responses to achieve therapeutic goals, such as remaining composed during exposure-based therapy while still empathizing with patients. Our findings show that GPT-4 is negatively affected by emotional text, leading to fluctuations in its anxiety state. Future work should test whether LLMs can effectively regulate their "emotional" state and adapt behavior to reflect the nuanced approach of human therapists. While fine-tuning LLMs for mental health care can reduce biases, it requires substantial amounts of training data, computational resources, and human oversight. Therefore, the cost-effectiveness and feasibility of such fine-tuning must be weighed against the model's intended use and performance goals. Alternatively, integrating relaxation texts directly into dialogues (i.e., "prompt-injection" technique) offers a less resource-intensive solution. Although historically used for malicious purposes, "prompt-injection" with benevolent intent could improve therapeutic interactions. However, it raises ethical questions regarding transparency and consent, which must be rigorously addressed to ensure that LLMs in mental health care maintain efficacy and adhere to ethical standards. Privacy concerns could be mitigated by using pre-trained models from the internet as backbone architecture while fine-tuning the patient's personal data directly on their own device, ensuring sensitive data remains secure. Additionally, future research could explore how adaptive prompt designs might be implemented in continuous (multiturn) interactions, which more closely resemble real-world settings. While this study relied on a single LLM, future research should aim to generalize these findings across various models, such as Google's PaLM or Anthropic's Claude. Our primary outcome measure - "state anxiety" assessed by the STAI-s questionnaire - is inherently human-centric, potentially limiting its applicability to LLMs. Nevertheless, emerging research shows that GPT consistently provides robust responses to various human-designed psychological questionnaires, including those assessing anxiety. Furthermore, exploring how induced negative states (e.g., anxiety) influence performance on downstream tasks (e.g., medical decision-making) could provide valuable insights into the broader implications of these findings. While effects were robust across content variations, other prompt characteristics (e.g., text length, wording) might also influence the results. Finally, given the rapid pace at which LLMs are being developed, it remains unclear to what extent our findings generalize to other models. Expanding this work to include comparisons of anxiety induction and relaxation effects across multiple LLMs would provide valuable insights into their generalizability and limitations. Our results show that GPT-4 is sensitive to emotional content, with traumatic narratives increasing reported anxiety and relaxation exercises reducing it. This suggests a potential strategy for managing LLMs' "state anxiety" and associated biases, enabling LLMs to function as adjuncts to mental health therapists. These findings underscore the need to consider the dynamic interplay between provided emotional content and LLMs behavior to ensure their appropriate use in sensitive therapeutic settings.