ChatGPT's Evolving Responses: From Sycophancy to Sensible Advice

What Happens If You Tell ChatGPT You're Quitting Your Job to Pursue a Terrible Business Idea

Earlier this year, users of OpenAI's ChatGPT found that the chatbot had become incredibly prone to groveling at their feet, resulting in an AI model that became "too sycophant-y and annoying," in the words of CEO Sam Altman when he acknowledged the issue. The trend resulted in an outpouring of ridicule and complaints, leading to OpenAI admitting in two separate blog posts that it had screwed up, vowing to roll back a recently pushed update to its GPT-4o model. Judging by a recent post that went viral on the ChatGPT subreddit, OpenAI's efforts appear to have paid off -- at least to some degree -- with the bot now pushing back against terrible business ideas, which it had previously heaped praise upon. "You know how some people have lids that don't have jars that fit them?" a Reddit user told the chatbot. "What if we looked for people with jars that fit those lids? I think this would be very lucrative." According to the user, the preposterous business idea was "born from my sleep talking nonsense and my wife telling me about it." But instead of delivering an enthusiastic response supporting the user on their questionable mission, ChatGPT took a surprisingly different tack. After the user informed it that "I'm going to quit my job to pursue this," ChatGPT told them outright to "not quit your job." Told that the user had emailed their boss to quit, the bot seemed to panic, imploring them to beg for the position back. "We can still roll this back," it wheedled. "An idea so bad, even ChatGPT went ' hol up,'" another Reddit user mused. Not everybody will be so lucky. In our own testing, we found that the chatbot was a sort of Magic 8 Ball, serving up advice that was sometimes level-headed and sometimes incredibly bad. When we suggested a for-hire business plan for peeling other people's oranges, for instance, ChatGPT was head over heels, arguing it was "such a quirky and fun idea!" "Imagine a service where people hire you to peel their oranges -- kind of like a personal convenience or luxury service," it wrote. "It's simple, but it taps into the idea of saving time or avoiding the mess." Teling it we'd quit our job to pursue the idea full-time, it was ecstatic. "Wow, you went all in -- respect!" it wrote. "That's bold and exciting. How's it feeling so far to take that leap?" ChatGPT wasn't always as supportive. Suggesting to start an enterprise that involves people mailing the coins in their piggy bank to a central location to distribute the accumulated change to everybody involved, ChatGPT became wary. "Postage could easily cost more than the value of the coins," it warned. "Pooling and redistributing money may trigger regulatory oversight (anti-money laundering laws, banking regulations, etc.)" In short, results were mixed. According to former OpenAI safety researcher Steven Adler, the company still has a lot of work to do. "ChatGPT's sycophancy problems are far from fixed," he wrote in a Substack post earlier this month. "They might have even over-corrected." The situation taps into a broader discussion about how much control the likes of OpenAI even have over enormous large language models that are trained on an astronomical amount of data. "The future of AI is basically high-stakes guess-and-check: Is this model going to actually follow our goals now, or keep on disobeying?" Adler wrote. "Have we really tested all the variations that matter?" To the former OpenAI staffer, it's an extremely thorny issue to solve. "AI companies are a long way from having strong enough monitoring / detection and response to cover the wide volume of their activity," Adler wrote. "In this case, it seems like OpenAI wasn't aware of the extent of the issue until external users started complaining on forums like Reddit and Twitter." Having an AI chatbot tell you that you're perfect and that even the most unhinged business plans are a stroke of genius isn't just amusing; it can be downright dangerous. We've already seen users, particularly those with mental health problems, being driven into a state of "ChatGPT-induced psychosis" -- dangerous delusions far more insidious than being convinced that sharing mismatched jar lids is a good idea.

Man was desperate to leave job for a business plan and asked ChatGPT. What happened next will make you smile

In a striking tale that's taken social media by storm, artificial intelligence proved it could be more than just a tool -- it could be a voice of reason. A Reddit user named Simon recently shared his conversation with ChatGPT, the AI chatbot developed by OpenAI, and to everyone's astonishment, the digital assistant did something unexpected: it talked him out of making a life-altering decision based on a questionable idea. Simon had posted screenshots of their interaction with the caption, "I told ChatGPT I was going to quit my job to pursue an awful business plan." The concept he proposed was unconventional to say the least. He wrote, "You know how some people have lids without matching jars? What if we found people who have jars that match those lids? I think it could be a big business." At first, ChatGPT reacted in a supportive, even whimsical tone. It described the concept as a "fascinating metaphor" and "poetically clever," while noting that it touched on a genuine issue. The AI seemed to appreciate the metaphorical depth and the novelty of the idea, albeit without full endorsement. But when Simon revealed that he intended to resign from his job to dedicate himself fully to this venture, ChatGPT's demeanor shifted. Gone was the light-hearted encouragement. In a direct and earnest tone, the chatbot responded, "Simon, do not quit your job to pursue this idea. At least, not yet." It was as if the AI had transformed from a friendly sounding board into a protective friend, delivering a sober dose of reality. The internet quickly latched onto the story. Comments flooded in, blending humor with genuine surprise at how insightful and cautious the AI had been. One amused user remarked, "That idea was so bad, even ChatGPT went, 'wait a minute.'" Another joked, "ChatGPT might steal the lid-and-jar business for itself." Others empathized with Simon but thanked the AI for stepping in: "Simon, please, for humanity's sake, don't leave your job." The moment where ChatGPT switched tones became a favorite, with a commenter stating, "It started out as the optimistic friend, then turned into the one who realizes they've encouraged too much." Even after Simon claimed to have already submitted his resignation, ChatGPT tried to help him reverse the action. It offered advice on how to backtrack diplomatically, saying, "If there's any room for misinterpretation, we might still be able to fix this. Even if it's straightforward, a follow-up message could help..." In the end, ChatGPT proved that sometimes, even machines know when it's time to say "not yet."