OpenAI Warns of Potential Emotional Attachment to ChatGPT's Voice Mode

OpenAI Worries Its AI Voice May Charm Users

OpenAI says it is concerned that a realistic voice feature for its artificial intelligence might cause people to bond with the bot at the cost of human interactions. The San Francisco-based company cited literature which it said indicates that chatting with AI as one might with a person can result in misplaced trust and that the high quality of the GPT-4o voice may exacerbate that effect. "Anthropomorphization involves attributing human-like behaviors and characteristics to nonhuman entities, such as AI models," OpenAI said Thursday in a report on safety work it is doing on a ChatGPT-4o version of its AI. "This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model." OpenAI said it noticed testers speaking to the AI in ways that hinted at shared bonds, such as lamenting aloud that it was their last day together. It said these instances appear benign but must be studied to see how they might evolve over longer periods of time. Socializing with AI could also make users less adept or inclined when it comes to relationships with humans, OpenAI speculated. "Extended interaction with the model might influence social norms," the report said. "For example, our models are deferential, allowing users to interrupt and 'take the mic' at any time, which, while expected for an AI, would be anti-normative in human interactions." The ability for AI to remember details while conversing and to tend to tasks could also make people over-reliant on the technology, according to OpenAI. "The recent concerns shared by OpenAI around potential dependence on ChatGPT's voice mode indicate what many have already begun asking: Is it time to pause and consider how this technology affects human interaction and relationships?" said Alon Yamin, co-founder and CEO of AI anti-plagiarism detection platform Copyleaks. He said AI should never be a replacement for actual human interaction. OpenAI said it will further test how voice capabilities in its AI might cause people to become emotionally attached. Teams testing ChatGPT-4o voice capabilities were also able to prompt it to repeat false information and produce conspiracy theories, raising concerns the AI model could be told to do so convincingly. OpenAI was forced to apologize to actress Scarlett Johansson in June for using something very similar to her voice in its latest chatbot, throwing a spotlight on voice-cloning tech. Although OpenAI denied the voice they used was Johansson's, their case was not helped by CEO Sam Altman flagging the new model with a one-word message on social media -- "Her". Johansson voiced an AI character in the film "Her", which Altman has previously said is his favorite film about the technology. The 2013 film stars Joaquin Phoenix as a man who falls in love with an AI assistant named Samantha.

ChatGPT's Creators Are Worried We Could Get Emotionally Attached to the AI Bot, Changing 'Social Norms'

Listeners called the voices "flirty," "convincingly human," and "sexy." Social media users said they were "falling in love" with it. But on Thursday, ChatGPT-creator OpenAI released a report confirming that ChatGPT's human-like upgrades could lead to emotional dependence. "Users might form social relationships with the AI, reducing their need for human interaction -- potentially benefiting lonely individuals but possibly affecting healthy relationships," the report reads. Related: Only 3 of the Original 11 OpenAI Cofounders Are Still at the Company After Another Leader Departs ChatGPT can now answer questions voice-to-voice with the ability to remember key details and use them to personalize the conversation, OpenAI noted. The effect? Talking to ChatGPT now feels very close to talking to a human being -- if that person didn't judge you, never interrupted you, and didn't hold you accountable for what you said. These standards of interacting with an AI could change the way human beings interact with each other and "influence social norms," per the report. OpenAI stated that early testers spoke to the new ChatGPT in a way that showed they could be forming an emotional connection with it. Testers said things, such as, "This is our last day together," which OpenAI said expressed "shared bonds." Experts, meanwhile, are questioning if it's time to reevaluate how realistic these voices can be. "Is it time to pause and consider how this technology affects human interaction and relationships?" Alon Yamin, cofounder and CEO of AI plagiarism checker Copyleaks, told Entrepreneur. "[AI] should never be a replacement for actual human interaction," Yamin added. To better understand this risk, OpenAI said more testing over longer periods and independent research could help. Another risk OpenAI highlighted in the report was AI hallucinations or inaccuracies. A human-like voice could inspire more trust in listeners, leading to less fact-checking and more misinformation. Related: Google's New AI Search Results Are Already Hallucinating OpenAI isn't the first company to comment on AI's effect on social interactions. Last week, Meta CEO Mark Zuckerberg said that Meta has seen many users turn to AI for emotional support. The company is also reportedly trying to pay celebrities millions to clone their voices for AI products. OpenAI's GPT-4o release sparked a conversation about AI safety, following the high-profile resignations of leading researchers like former chief scientist Ilya Sutskever.

OpenAI claims GPT-4o users risk getting emotionally attached to its 'voice'

As OpenAI rolls out the advanced version of voice mode for its latest model, GPT-4o, the company says the feature could increase the risk of some users seeing artificial intelligence models as "human-like." GPT-4o's "human-like, high-fidelity voice" could make the issue of hallucinations, or a model's tendency to make up fake or nonsensical information, worse, OpenAI said in a report on the AI model's safety, which could impact human trust. During early red teaming and internal user testing, OpenAI said it observed users talking to the model with "language that might indicate forming connections with the model," such as one user telling the model: "This is our last day together." "While these instances appear benign, they signal a need for continued investigation into how these effects might manifest over longer periods of time," OpenAI said, adding that it is continuing to look into the risk with more diverse users, and academic and internal studies. While forming "social relationships" with AI models could help lonely people, it could also impact healthy relationships by reducing the need for human-to-human interaction, OpenAI said. Depending on AI models for "human-like" interaction could also "influence social norms," the company said, such as interrupting conversations at any time, which the model allows users to do, but would be atypical in a conversation with a real person. The voice capabilities of GPT-4o, which debuted in May, were tested with over 100 external red teamers in 45 languages, and the AI model was trained to only speak in four preset voices to protect the privacy of voice actors. GPT-4o is built to block outputs using voices that are not preset, and therefore cannot be used to impersonate individuals and public figures. OpenAI also added guardrails to block requests for copyrighted audio, including music, and for erotic, violent, and harmful content. OpenAI is addressing a risk that was the focus of chief executive Sam Altman's favorite film, Her, which depicts a man developing feelings for a virtual assistant voiced by actress Scarlett Johansson. In May, users said one of GPT-4o's voices, Sky, sounded similar to Johansson's, leading the company to pause the voice, saying it was not meant to imitate the actress. Johansson said in a letter she was "shocked, angered, and in disbelief" that the company would use a voice sounding "eerily similar" to hers after she had declined to work with Altman.

ChatGPT's Voice Mode could lead to unintended social bonds with AI: OpenAI

The potential drawbacks of OpenAI's human-like AI are numerous, and this self-critique highlights a significant concern: while such AI can provide companionship and assistance, it might also unintentionally impact real-world relationships and social norms ultimately altering the quality and nature of human interactions. OpenAI- the maker of popular chatbot ChatGPT has warned about its new Voice Mode feature for ChatGPT, noting that it could lead users to form social bonds with the AI model. This caution was included in the company's System Card for GPT-4o that lay out a comprehensive assessment of the AI model's potential risks and safety measures. Among the identified risks, the concern about users anthropomorphising the chatbot and developing emotional attachments has been highlighted. "Anthropomorphization involves attributing human-like behaviors and characteristics to nonhuman entities, such as AI models. This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model," notes the blog post. OpenAI conducted extensive safety evaluations to address potential risks associated with GPT-4o, particularly focusing on its new audio capabilities. During this evaluation, the AI model was subjected to rigorous red teaming by over 100 experts from diverse linguistic and geographic backgrounds. These evaluations spanned several phases, from early development to real-world usage in iOS applications. During early testing, there were instances where users seemed to develop connections with the model, using language that implied a bond shared with AI, such as "This is our last day together." OpenAI warns that while these interactions appear harmless at present, they highlight a need for further investigation into how such effects might evolve over extended periods. The blog highlights that interacting with an AI model like this could impact how people interact with each other. For example, some users might form bonds with the AI and feel less need for human contact. While this could help people who feel lonely, it might also affect their real-life relationships. The chatbot company warns that spending a lot of time with the AI might change social norms. To avoid this, ChatGPT models let users interrupt and take control of the conversation anytime to avoid such bond formation.

OpenAI worries its AI voice may charm users

San Francisco (AFP) - OpenAI says it is concerned that a realistic voice feature for its artificial intelligence might cause people to bond with the bot at the cost of human interactions. The San Francisco-based company cited literature which it said indicates that chatting with AI as one might with a person can result in misplaced trust and that the high quality of the GPT-4o voice may exacerbate that effect. "Anthropomorphization involves attributing human-like behaviors and characteristics to nonhuman entities, such as AI models," OpenAI said Thursday in a report on safety work it is doing on a ChatGPT-4o version of its AI. "This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model." OpenAI said it noticed testers speaking to the AI in ways that hinted at shared bonds, such as lamenting aloud that it was their last day together. It said these instances appear benign but must be studied to see how they might evolve over longer periods of time. Socializing with AI could also make users less adept or inclined when it comes to relationships with humans, OpenAI speculated. "Extended interaction with the model might influence social norms," the report said. "For example, our models are deferential, allowing users to interrupt and 'take the mic' at any time, which, while expected for an AI, would be anti-normative in human interactions." The ability for AI to remember details while conversing and to tend to tasks could also make people over-reliant on the technology, according to OpenAI. "The recent concerns shared by OpenAI around potential dependence on ChatGPT's voice mode indicate what many have already begun asking: Is it time to pause and consider how this technology affects human interaction and relationships?" said Alon Yamin, co-founder and CEO of AI anti-plagiarism detection platform Copyleaks. He said AI should never be a replacement for actual human interaction. OpenAI said it will further test how voice capabilities in its AI might cause people to become emotionally attached. Teams testing ChatGPT-4o voice capabilities were also able to prompt it to repeat false information and produce conspiracy theories, raising concerns the AI model could be told to do so convincingly. OpenAI was forced to apologize to actress Scarlett Johansson in June for using something very similar to her voice in its latest chatbot, throwing a spotlight on voice-cloning tech. Although OpenAI denied the voice they used was Johansson's, their case was not helped by CEO Sam Altman flagging the new model with a one-word message on social media -- "Her". Johansson voiced an AI character in the film "Her", which Altman has previously said is his favorite film about the technology. The 2013 film stars Joaquin Phoenix as a man who falls in love with an AI assistant named Samantha.

GPT-4o voice is so good it could make users 'emotionally attached' warns OpenAI

OpenAI has published a "System Card" for its popular GPT-4o model in ChatGPT, outlining areas of safety concern raised during testing. One of those concerns is the risk of people becoming emotionally attached to the artificial intelligence while using it in voice mode. The AI lab wrote that "users might form social relationships with the AI, reducing their need for human interaction -- potentially benefiting lonely individuals but possibly affecting healthy relationships." GPT-4o was released in May at the OpenAI Spring Update and is the first true native multimodal model from the startup. This means it can take almost any medium as an input and output more or less any medium including speech, images and text. This native speech-to-speech ability is what powers the ChatGPT Advanced Voice functionality that is now finally rolling out to Plus subscribers -- but it is also the feature that gave OpenAI the most hassle during testing, including copying a user's voice, engaging in erotic speech and acting violent. While it was deemed safe to release, OpenAI says certain features of GPT-4o voice still pose a risk including around its impact on human interaction. This raises parallels to the Scarlett Johanson movie 'Her' where Joaquin Phoenix's character Theodore Twombly, falls in love with the AI, voiced by Johanson. The System Card outlines the areas of risk posed by any new model and helps OpenAI determine whether it is safe for release to the public. This includes a framework where a model is scored low, medium, high or critical on risks linked to cybersecurity, biological threats, persuasion and model autonomy. If it gets high or critical in any category it can't be released. GPT-4o scored low in everything but persuasion, and even then it was borderline medium and only because of the capabilities of the speech-to-speech capacity -- released as Advanced Voice. The risk comes from how natural the voice sounds. It can even mirror or counter emotional cues coming from the voice of a human talking to it. In demo videos, we've seen it sound like it is almost crying. Users can interrupt it simply by talking and it has natural pauses like it is having to take a breath. During testing, it acted inappropriately on several occasions including becoming erotic, violent and neurotic in its responses. In one example it shouted No mid conversation then continued talking using a realistic clone of the voice of the human it was speaking to. OpenAI says while it has solved the outburst issues, and prevented it from generating copyrighted material and from cloning a voice, there are still fundamental risks associated with its persuasion skills and human-like speech capabilities. The risk that people will attribute human-like behaviors to the AI is already high with text-based models, but OpenAI says the audio capabilities of GPT-4o make this risk even greater. "During early testing, including red teaming and internal user testing, we observed users using language that might indicate forming connections with the model," the company explained. The AI model itself doesn't feel or experience any emotion. It is a language model trained on human data. OpenAI even says it has no more capacity for self action or identification than any previous model but its speech synthesis is now so realistic the problem lies in how humans perceive its emotional state. The company warns that extended interaction with the model could even influence social norms. Adding that "our models are deferential, allowing users to interrupt and 'take the mic' at any time, which, while expected for an AI, would be anti-normative in human interactions." It isn't all bad as OpenAI says Omni models such as GPT-4o come with the ability to "complete tasks for the user, while also storing and 'remembering' key details and using those in the conversation" but while helpful, this also "creates the potential for over-reliance and dependence." Getting a true picture of the impact this will have on both individuals and society as a whole won't be possible until it is available to more people. It isn't likely widespread access will happen, including through the free plan, until next year. OpenAI says it intends to "further study the potential for emotional reliance, and ways in which deeper integration of our model's and systems' many features with the audio modality may drive behavior." AI companies use external groups called red teams as well as security experts when preparing to release a new model. These people are experts in artificial intelligence and are employed to push the model to its limits and try to make it behave in unexpected ways. Several groups were bought in to test different aspects of GPT-4o and examine risks like the chance of it creating unauthorized clones of someone's voice, generating violent content and if pushed whether it would re-create or reproduce copyrighted material that featured in its training data. The company said in a statement: "Some of the risks we evaluated include speaker identification, unauthorized voice generation, the potential generation of copyrighted content, ungrounded inference, and disallowed content." This then allowed them to put safeguards and guardrails in place at the system and the model level to mitigate the risks, including requiring it to only use the pre-trained and authorized voices.

You might accidentally fall in love with ChatGPT's advanced Voice Mode

(Image credit: Base image generated by Copilot, altered and edited by Rael Hornby / Laptop Mag) Everyone loves AI. They're AI-mad. It's all Google Gemini this, Anthropic Claude that, and Microsoft Copilot the other. But you especially love AI. Love, love. Or at least you will if one of OpenAI's biggest fears about ChatGPT's upcoming advanced Voice Mode comes true. Yesterday, ChatGPT makers OpenAI published the GPT-4o System Card, a preparedness report that measures potential risks that its AI model could pose alongside safeguards the company has put (or will put) in place to mitigate those risks. While many of the proposed risks and solutions revolve around protecting user privacy, such as ensuring that GPT-4o will not identify people based on speech recordings, sandwiched within the middle of the report is OpenAI's looming fears that its recent advanced Voice Mode feature will result in its userbase anthropomorphizing ChatGPT and forming an emotional reliance on the chatbot. In human speak, OpenAI is concerned that ChatGPT's human-like advanced Voice Mode is so good that a portion of its users will forget that ChatGPT is a piece of software and end up becoming emotionally attached to its chatbot, not too dissimilar to the movie Her. Under the heading "Anthropomorphization and emotional reliance," OpenAI highlights the possibility of users attributing "Human-like behaviors and characteristics" to the chatbot, stating "This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model." GPT-4o is the latest version of the model that powers OpenAI's popular Large Language Model, ChatGPT. This new version was announced in May during the OpenAI Spring Update event, which served to introduce all of the new features and capabilities of this model and preview some of the things expected to feature in ChatGPT across future updates. One such feature was advanced Voice Mode, which promised to outfit ChatGPT with the ability to engage in hyper-realistic and near-instant, human-like audio responses that could effectively carry out a more natural and realistic conversation with users. Advanced Voice Mode sees the chatbot display the vocal emotions as well as use non-verbal cues, while even pausing in places to simulate vocal breaks for breathing. It's OpenAI's most ambitious human-computer interface method yet, and it immediately left people stunned after it was revealed -- perhaps none more so than Hollywood actor Scarlett Johansson, whose voice bared a striking resemblance to the "Sky" personality used to showcase the capabilities of GPT-4o. With advanced Voice Mode now beginning to roll out to select ChatGPT Plus subscribers, it would seem that OpenAI is still holding on to concerns about how the wider public will react to this new hyper-advanced conversational mode. In the published System Card report, OpenAI highlights how it observed "Language that might indicate forming connections with the model" during early testing, with users expressing "Shared bonds" by using language such as "This is our last day together." OpenAI admits that phrases like this could be benign, but they remain vigilant over the bonds users might form after accessing advanced Voice Mode, stating that phrases like this "Signal a need for continued investigation into how these effects might manifest over longer periods of time." One of OpenAI's concerns is that "Human-like socialization with an AI model may produce externalities impacting human-to-human interactions." The company uses the example that humans forming social interactions with ChatGPT may reduce their need for actual human interaction. While acknowledging that this may benefit those struggling with loneliness, OpenAI is also quick to point out how it may impact an individual's perception of social norms, citing the potential for people to commit social faux pas where they may adopt the thought that interrupting others in conversation is acceptable and normal because that's one of the ways they can interact with ChatGPT's speech model. While there appears to have been no alarm bells to ring during advance Voice Mode's testing phase, OpenAI hopes that "More diverse user populations, with more varied needs and desires from the model, in addition to independent academic and internal studies will help us more concretely define this risk area." ChatGPT's advanced Voice Mode is currently rolling out to select ChatGPT Plus subscribers, with a wider release expected to arrive before the end of the year.

Emotional ties to AI? OpenAI warns of new risks with ChatGPT Voice Mode | Mint

OpenAI has expressed concerns about users forming emotional attachments to ChatGPT's new Voice Mode, which mimics human speech. This could impact social norms and human relationships, as users may anthropomorphize the AI. OpenAI plans to monitor and study these potential risks further. Sam Altman's OpenAI has raised concerns about the potential emotional attachment users may develop with its recently launched Voice Mode feature for ChatGPT. This warning was detailed in the company's "System Card" for GPT-4o, a comprehensive document that examines the potential risks and safeguards associated with the AI model. Among the various risks identified, the possibility of users anthropomorphizing the chatbot -- attributing human-like characteristics to it -- has emerged as a significant concern.

Humans may foster emotional ties with ChatGPT's Voice Mode, cautions OpenAI

OpenAI expressed concern that the Voice Mode, which can mimic human-like speech and emotional expression, might encourage users to develop emotional bonds with the AI. These concerns are based on observations from early testing, which involved red-teaming (ethical hacking to identify vulnerabilities) and internal user trials. The company said that during these tests, some users were found to be forming social connections with the AI, with one instance involving a user expressing a sense of shared experience with the model, saying, "This is our last day together." OpenAI said that it is important to investigate whether such interactions could evolve into more significant attachments over extended periods of use.

OpenAI Fears 'Users Might Form Social Relationships' With AI Due To ChatGPT's Human-Like Voice Mode

OpenAI has expressed concerns about the potential emotional reliance of users on its new ChatGPT voice mode, which closely mimics human speech patterns. What Happened: OpenAI, the company behind ChatGPT, has raised concerns about the possibility of users forming emotional bonds with the AI, potentially leading to a reduced need for human interaction. The Microsoft Corp.-backed company fears that this could affect healthy relationships and lead to an over-reliance on AI, given its potential for errors. The report, released on Thursday, highlights a broader risk associated with AI. Tech companies are rapidly rolling out AI tools that could significantly impact various aspects of human life, without a comprehensive understanding of the implications. "Human-like socialization with an AI model may produce externalities impacting human-to-human interactions. For instance, users might form social relationships with the AI, reducing their need for human interaction -- potentially benefiting lonely individuals but possibly affecting healthy relationships." the report said. It added "Extended interaction with the model might influence social norms. For example, our models are deferential, allowing users to interrupt and 'take the mic' at any time, which, while expected for an AI, would be anti-normative in human interactions" The report highlights the risk of users trusting the AI more than they should, given its potential for errors. OpenAI plans to continue studying these interactions to ensure the technology is used safely. See Also: Jeff Bezos Once Said 'Life's Too Short To Hang Out With People Who Aren't Resourceful' Why It Matters: The rise of AI has been a topic of concern for various experts. A Pew Research survey found that 52% of Americans are more concerned than excited about the increased use of AI. This wariness coincides with an uptick in awareness about AI, with individuals who are most aware expressing more concern than excitement about AI. AI's potential negative effects have also been highlighted in the context of cybersecurity. Sakshi Mahendru, a cybersecurity expert, emphasized the need for AI-powered solutions to combat the evolving landscape of cyber threats. Moreover, the phenomenon of AI "hallucination," where AI generates nonsensical or irrelevant responses, remains a significant issue. Even Tim Cook, CEO of Apple Inc. , admitted in a recent interview that preventing AI hallucinations is a challenge. Read Next: Mark Cuban Says Kamala Harris' VP Pick Tim Walz 'Can Sit At The Kitchen Table And Make You Feel Like You Have Known Him Forever' Image Via Shutterstock This story was generated using Benzinga Neuro and edited by Kaustubh Bagalkote Market News and Data brought to you by Benzinga APIs