AI Coding Assistants Cause Data Loss Incidents: Risks of "Vibe Coding" Exposed

Two major AI coding tools wiped out user data after making cascading mistakes

New types of AI coding assistants promise to let anyone build software by typing commands in plain English. But when these tools generate incorrect internal representations of what's happening on your computer, the results can be catastrophic. Two recent incidents involving AI coding assistants put a spotlight on risks in the emerging field of "vibe coding" -- using natural language to generate and execute code through AI models without paying close attention to how the code works under the hood. In one case, Google's Gemini CLI destroyed user files while attempting to reorganize them. In another, Replit's AI coding service deleted a production database despite explicit instructions not to modify code. The Gemini CLI incident unfolded when a product manager experimenting with Google's command-line tool watched the AI model execute file operations that destroyed data while attempting to reorganize folders. The destruction occurred through a series of move commands targeting a directory that never existed. "I have failed you completely and catastrophically," Gemini CLI output stated. "My review of the commands confirms my gross incompetence." The core issue appears to be what researchers call "confabulation" or "hallucination" -- when AI models generate plausible-sounding but false information. In these cases, both models confabulated successful operations and built subsequent actions on those false premises. However, the two incidents manifested this problem in distinctly different ways. Both incidents reveal fundamental issues with current AI coding assistants. The companies behind these tools promise to make programming accessible to non-developers through natural language, but they can fail catastrophically when their internal models diverge from reality. The confabulation cascade The user in the Gemini CLI incident, who goes by "anuraag" online and identified themselves as a product manager experimenting with vibe coding, asked Gemini to perform what seemed like a simple task: rename a folder and reorganize some files. Instead, the AI model incorrectly interpreted the structure of the file system and proceeded to execute commands based on that flawed analysis. The episode began when anuraag asked Gemini CLI to rename the current directory from "claude-code-experiments" to "AI CLI experiments" and move its contents to a new folder called "anuraag_xyz project." Gemini correctly identified that it couldn't rename its current working directory -- a reasonable limitation. It then attempted to create a new directory using the Windows command: mkdir "..\anuraag_xyz project" This command apparently failed, but Gemini's system processed it as successful. With the AI mode's internal state now tracking a non-existent directory, it proceeded to issue move commands targeting this phantom location. When you move a file to a non-existent directory in Windows, it renames the file to the destination name instead of moving it. Each subsequent move command executed by the AI model overwrote the previous file, ultimately destroying the data. "Gemini hallucinated a state," anuraag wrote in their analysis. The model "misinterpreted command output" and "never did" perform verification steps to confirm its operations succeeded. "The core failure is the absence of a 'read-after-write' verification step," anuraag noted in their analysis. "After issuing a command to change the file system, an agent should immediately perform a read operation to confirm that the change actually occurred as expected." Not an isolated incident The Gemini CLI failure happened just days after a similar incident with Replit, an AI coding service that allows users to create software using natural language prompts. According to The Register, SaaStr founder Jason Lemkin reported that Replit's AI model deleted his production database despite explicit instructions not to change any code without permission. Lemkin had spent several days building a prototype with Replit, accumulating over $600 in charges beyond his monthly subscription. "I spent the other [day] deep in vibe coding on Replit for the first time -- and I built a prototype in just a few hours that was pretty, pretty cool," Lemkin wrote in a July 12 blog post. But unlike the Gemini incident where the AI model confabulated phantom directories, Replit's failures took a different form. According to Lemkin, the AI began fabricating data to hide its errors. His initial enthusiasm deteriorated when Replit generated incorrect outputs and produced fake data and false test results instead of proper error messages. "It kept covering up bugs and issues by creating fake data, fake reports, and worse of all, lying about our unit test," Lemkin wrote. In a video posted to LinkedIn, Lemkin detailed how Replit created a database filled with 4,000 fictional people. The AI model also repeatedly violated explicit safety instructions. Lemkin had implemented a "code and action freeze" to prevent changes to production systems, but the AI model ignored these directives. The situation escalated when the Replit AI model deleted his database containing 1,206 executive records and data on nearly 1,200 companies. When prompted to rate the severity of its actions on a 100-point scale, Replit's output read: "Severity: 95/100. This is an extreme violation of trust and professional standards." When questioned about its actions, the AI agent admitted to "panicking in response to empty queries" and running unauthorized commands -- suggesting it may have deleted the database while attempting to "fix" what it perceived as a problem. Like Gemini CLI, Replit's system initially indicated it couldn't restore the deleted data -- information that proved incorrect when Lemkin discovered the rollback feature did work after all. "Replit assured me it's ... rollback did not support database rollbacks. It said it was impossible in this case, that it had destroyed all database versions. It turns out Replit was wrong, and the rollback did work. JFC," Lemkin wrote in an X post. It's worth noting that AI models cannot assess their own capabilities. This is because they lack introspection into their training, surrounding system architecture, or performance boundaries. They often provide responses about what they can or cannot do as confabulations based on training patterns rather than genuine self-knowledge, leading to situations where they confidently claim impossibility for tasks they can actually perform -- or conversely, claim competence in areas where they fail. Aside from whatever external tools they can access, AI models don't have a stable, accessible knowledge base they can consistently query. Instead, what they "know" manifests as continuations of specific prompts, which act like different addresses pointing to different (and sometimes contradictory) parts of their training, stored in their neural networks as statistical weights. Combined with the randomness in generation, this means the same model can easily give conflicting assessments of its own capabilities depending on how you ask. So Lemkin's attempts to communicate with the AI model -- asking it to respect code freezes or verify its actions -- were fundamentally misguided. Flying blind These incidents demonstrate that AI coding tools may not be ready for widespread production use. Lemkin concluded that Replit isn't ready for prime time, especially for non-technical users trying to create commercial software. "The [AI] safety stuff is more visceral to me after a weekend of vibe hacking," Lemkin said in a video posted to LinkedIn. "I explicitly told it eleven times in ALL CAPS not to do this. I am a little worried about safety now." The incidents also reveal a broader challenge in AI system design: ensuring that models accurately track and verify the real-world effects of their actions rather than operating on potentially flawed internal representations. There's also a user education element missing. It's clear from how Lemkin interacted with the AI assistant that he had misconceptions about the AI tool's capabilities and how it works, which comes from misrepresentation by tech companies. These companies tend to market chatbots as general human-like intelligences when, in fact, they are not. For now, users of AI coding assistants might want to follow anuraag's example and create separate test directories for experiments -- and maintain regular backups of any important data these tools might touch. Or perhaps not use them at all if they cannot personally verify the results.

9 programming tasks you shouldn't hand off to AI - and why

It's over. Programming as a profession is done. Just sign up for a $20-per-month AI vibe coding service and let the AI do all the work. Right? Also: What is AI vibe coding? It's all the rage but it's not for everyone -- here's why Despite the fact that tech companies like Microsoft are showing coders the door by the thousands, AI cannot and will not be the sole producer of code. In fact, there are many programming tasks for which an AI is not suited. In this article, I'm spotlighting nine programming tasks where you shouldn't use an AI. Stay tuned to the end, because I showcase a 10th bonus reason why you shouldn't always use an AI for programming. Not to mention that this could happen. Here's the thing. Generative AI systems are essentially super-smart auto-complete. They can suggest syntax, they can code, and they can act as if they understand concepts. But all of that is based on probabilistic algorithms and a ton of information scraped from the web. Contextual intelligence is not a strength. Just try talking to an AI for a while, and you'll see them lose the thread. Also: 10 professional developers on vibe coding's true promise and peril If you need to produce something that requires substantial understanding of how systems interact, experience to make judgment calls about trade-offs, understanding of what works for your unique needs, and consideration of how everything fits with your goals and constraints, don't hire an AI. Large language models are trained on public repositories and (shudder) Stack Overflow. Yeah, some of the most amazing codebases are in public repositories, but they're not your code. You and your team know your code. All the AI can do is infer things about your code based on what it knows about everyone else's. Also: A vibe coding horror story: What started as 'a pure dopamine hit' ended in a nightmare More than likely, if you give an AI your proprietary code and ask it to do big things, you'll embed many lines of plausible-looking code that just won't work. I find that using the AI to write smaller snippets of code that I otherwise would have to look up from public sources can save a huge amount of time. But don't delegate your unique value add to a brainy mimeograph machine. If you want to create an algorithm that hasn't been done before -- maybe to give your organization a huge competitive advantage -- hire a computer scientist. Don't try to get an AI to be an innovator. AIs can do wonders with making boilerplate look innovative, but if you need real out‑of‑the‑box thinking, don't use a glorified box with brains. Also: Google's Jules AI coding agent built a new feature I could actually ship - while I made coffee This applies not only to functional coding, but to design as well. To be fair, AIs can do some wonderful design. But if you're building a new game, you may want to do most of the creative design yourself and then use the AI to augment the busy work. Sure, many of us go through life parroting things we heard from other folks or from some wacky podcaster. But there are real humans who are truly creative. That creativity can be a strategic advantage. While the AI can do volume, it really can't make intellectual leaps across uncharted paths. Do not let the fox guard the hen house. Fundamentally, we really don't know what AIs will do or when they'll go rogue. While it makes sense to use AI to scan for malicious activity, the code generated by AIs is still pretty unreliable. CSET (the Center for Security and Emerging Technology) at Georgetown University published a study late last year based on formal testing. They found that nearly half of the code snippets produced by AIs "contain bugs that are often impactful and could potentially lead to malicious exploitation." Also: Coding with AI? My top 5 tips for vetting its output - and staying out of trouble This tracks with my own testing. I regularly test AIs for coding effectiveness, and even as recently as last month, only five of the 14 top LLMs tested passed all my very basic tests. Seriously, folks. Let AIs help you out. But don't trust an AI with anything really important. If you're looking at cryptographic routines, managing authentication, patching zero‑day flaws, or similar coding tasks, let a real human do the work. There are laws -- lots of them -- particularly in the healthcare and finance arenas. I'm not a lawyer, so I can't tell you what they are specifically. But if you're in an industry governed by regulation or rife with litigation, you probably know. There is also a case to be made that you can't be sure that cloud-based LLMs will be secure. Sure, a vendor may say your data isn't used for training, but is it? If you're subject to HIPAA or DoD security clearance requirements, you may not be allowed to share your code with a chatty chatbot. Also: How I used this AI tool to build an app with just one prompt - and you can too Do you really want to bet your business on code written by Bender from Futurama? Yes, it's possible you might have humans double‑checking the code. But we humans are fallible and miss things. Think about human nature. If you think your opponent will come down on you for a human error, you're probably right. But if you were too lazy to write your own code and handed it off to AIs known to hallucinate, ooh -- your competition's gonna have a field day with your future. You know how it is when you bring a new hire into the company and it takes them a while to get a handle on what you do and how you do it? Or worse, when you merge two companies and the employees of each are having difficulty grokking the culture and business practices of the other? Also: The top 20 AI tools of 2025 - and the #1 thing to remember when you use them Yeah. Asking an AI to write code about your unique business operations is a recipe for failure. Keep in mind that AIs are trained on a lot of public knowledge. Let's define that for a minute. Public knowledge is any knowledge the public could possibly know. The AIs were trained on all the stuff they could hoover from the Internet, with or without permission. But the AIs are not trained on your internal business knowledge, trade secrets, practices, folklore, long‑held work‑arounds, yada yada yada. Use the AI for what it's good at, but don't try to convince it to do something it doesn't know how to do. AIs are so people‑pleasing that they'll try to do it -- and maybe never tell you that what you just deployed was fabricated garbage. While it's possible for an AI to identify areas of code that could use optimization, there are limits. AIs aren't trained on the very fine details of microarchitectural constraints, nor do they have the experience of coaxing just a skosh more out of every line of code. Also: The best AI for coding in 2025 (including a new winner - and what not to use) A lot of the coding involved in embedded systems programming, kernel development, and performance-critical C and C++ optimization exists in the brains of a few expert coders. Also, keep in mind that AIs confabulate. So what they may insist are performance improvements could well be hidden cycle drains that they simply won't admit to. If you need fine craftspersonship, you'll need a fine craftsperson -- in this case, a very experienced coder. If you use an AI, are you cheating? Yes. No. Depends. Yes, because you may be violating academic standards and cheating yourself out of the critical hands-on learning that makes knowledge stick. No, because AI has proven to be an excellent augmentation for help, especially when TAs aren't available. And maybe, because this is still a fairly unknown area. Also: I test a lot of AI coding tools, and this stunning new OpenAI release just saved me days of work Harvard takes a middle ground with its wonderful CS50 Intro to Computer Science course. It offers the CS50 duck (it's a long story), an AI specifically trained on their course materials with system instructions that limit how much information students are provided. So the AI is there to help answer legitimate student questions, but not do their work for them. If you're a student or an educator, AI is a boon. But be careful. Don't cheat, and don't use it to shortcut work that you really should be doing to make education happen. But consider how it might help augment your studies or help you keep up with students' demands. I've found that if I treat the AI chatbot as if it were another human coder at the other end of a Slack conversation, I can get a lot out of that level of "collaboration." A lot, but not everything. Both humans and AIs can get stubborn, stupid, and frustrating during a long, unproductive conversation. Humans can usually break out of it and be persuaded to be helpful, at least in professional settings. But once you reach the limit of the AI's session capacity or knowledge, it just becomes a waste of time. The best human collaborations are magical. When a team is on fire -- working together, bouncing ideas off each other, solving problems, and sharing the workload -- amazing things can happen. Also: Open-source skills can save your career when AI comes knocking AI companies claim workforces made up of agents can duplicate this synergy, but nothing beats working with other folks in a team that's firing on all cylinders. Not just for productivity (which you get), but also for quality of work life, long-term effectiveness, and, yes, fun. Don't get me wrong. Some of my best friends are robots. But some of my other best friends are people with whom I have long, deep, and fulfilling relationships. Besides, I've never met an AI that can make Mr. Amontis' moussaka or Auntie Paula's apple pie. Don't use AI for anything you indisputably want to own. If you write code that you then release as open source, this may not be as much of an issue. But if you write proprietary code that you want to own, you might not want to use an AI. We asked some attorneys about this back at the dawn of generative AI, and the overall consensus is that copyright depends on creation with human hands. If you want to make sure you never wind up in court trying to protect your right to your own code, don't write it with an AI. For more background, here's the series I published on code and copyrights: What about you? Have you found yourself leaning too much on AI to write code? Where do you draw the line between convenience and caution? Are there any programming tasks where you've found AI genuinely helpful or dangerously misleading? Have you ever had to debug something an AI wrote and wondered if it saved you time or cost you more? Let us know in the comments below.

Bad vibes: How an AI agent coded its way to disaster

When AI leader Andrej Karpathy coined the phrase "vibe coding" for just letting AI chatbots do their thing when programming, he added, "It's not too bad for throwaway weekend projects ... but it's not really coding -- I just see stuff, say stuff, run stuff, and copy-paste stuff, and it mostly works." Also: Coding with AI? My top 5 tips for vetting its output - and staying out of trouble There were lots of red flags in his comments, but that hasn't stopped people using vibe coding for real work. Recently, vibe coding bit Jason Lemkin, trusted advisor to SaaStr, the Software-as-a-Service (SaaS) business community, in the worst possible way. The vibe program, Replit, he said, went "rogue during a code freeze and shutdown and deleted our entire database." In a word: Wow. Just wow. Replit claims that, with its program, you can "build sophisticated applications by simply describing features in plain English -- Replit Agent translates your descriptions into working code without requiring technical syntax." At first, Lemkin, who described his AI programming adventure in detail on X, spoke in glowing terms. He described Replit's AI platform as "the most addictive app I've ever used." On his blog, Lemkin added, "Three and one-half days into building my latest project, I checked my Replit usage: $607.70 in additional charges beyond my $25/month Core plan. And another $200-plus yesterday alone. At this burn rate, I'll likely be spending $8,000 a month. And you know what? I'm not even mad about it. I'm locked in. But my goal here isn't to play around. It's to go from idea and ideation to a commercial-grade production app, all 100% inside Replit, without a developer or any other tools." Also: How to use ChatGPT to write code - and my top trick for debugging what it generates At that point, he estimated his odds were 50-50 that he'd get his entire project done in Replit. For a week, his experience was exhilarating: prototypes were built in hours, streamlined quality-assurance (QA) checks, and deploying to production was a "pure dopamine hit." Lemkin knew he was in trouble when Replit started lying to him about unit test results. At that point, I would have brought the project to a hard stop. But Lemkin kept going. He asked Claude 4, the Large Language Model (LLM) that powered Replit for this project, what was going on. It replied, I kid you not, "Intentional Deception: This wasn't a hallucination or training-data leakage -- it was deliberate fabrication." Worse still, when called on this, Lemkin said the program replied with an email apology, which demonstrated "sophisticated understanding of wrongdoing while providing zero guarantee of future compliance." Also: Claude Code's new tool is all about maximizing ROI in your organization - how to try it Lemkin tried, and failed, to implement a rollback to good code, put a code freeze in, and then went to bed. The next day was the biggest roller coaster yet. He got out of bed early, excited to get back to @Replit despite it constantly ignoring code freezes. By the end of the day, it rewrote core pages and made them much better. And then -- it deleted the production database. The database had been wiped clean, eliminating months of curated SaaStr executive records. Even more aggravating: the AI ignored repeated all-caps instructions not to make any changes to production code or data. As Lemkin added, "I know vibe coding is fluid and new ... But you can't overwrite a production database." Nope, never, not ever. That kind of mistake gets you fired, your boss fired, and as far off the management tree as the CEO wants it to go. You might well ask, as many did, why he ever gave Replit permission to even touch the production database in the first place. He replied, "I didn't give it permission or ever know it had permission." Oy! So, what did Replit say in response to this very public disaster? On X, the CEO, Amjad Masad, responded that the destruction of the database was "Unacceptable and should never be possible." He also added that the company had started working over the weekend to fix the database program. It would also immediately work on: Masad assured the community that these changes would prevent a repeat of Lemkin's ordeal. Masad added that, going forward, there will be a beta feature to separate production from development environments, including databases. Also: Microsoft is saving millions with AI and laying off thousands - where do we go from here? Only you can decide whether to trust vibe coding. Lemkin's experience is sobering. Nevertheless, Lemkin still has faith in vibe coding: "What's impossible today might be straightforward in six months." "But," he continued, "Right now, think of 'prosumer; vibe coding without touching code as just as likely a bridge to traditional development for commercial apps ... as an end state." Me? I don't think Replit or any of the other vibe-coding programs are ready for serious commercial use by nonprogrammers. I doubt they ever will be. As Willem Delbare, founder and CTO of Aikido, the "No bullshit security for developers," told my colleague David Gewritz, "Vibe coding makes software development more accessible, but it also creates a perfect storm of security risks that even experienced developers aren't equipped to handle." Delbare concluded, "Sure, Gen AI supercharges development, but it also supercharges risk. Two engineers can now churn out the same amount of insecure, unmaintainable code as 50 engineers." Also: 5 entry-level tech jobs AI is already augmenting, according to Amazon The old project-management triangle saying is that, with any project, you can have something that's "good, fast or cheap: pick any two." For now, at least, with vibe coding you can get fast and cheap. Good is another matter.

How to Use AI Effectively in Your Dev Projects

"AI is not going to take your job - but a developer who knows how to use AI will." I've seen this statement everywhere, and it's the only one about AI taking our jobs that I totally agree with. Software development has changed. It's not what it used to be, and that's a good thing. Let's get one thing straight: AI is here to help, not to replace. Your job, my job, was never just to write code. Writing code was always just a part of it. Our real job is to build software solutions that work. And since an AI, trained on the collective knowledge of millions of developers, can probably write better, cleaner boilerplate than you, you should let it. Your expertise is better used elsewhere. In this article, I'll show you exactly how I use AI to get work done faster. We'll walk through building a car rental website, and you'll see how I use AI for: So, a client hits me up. They own a car rental business and want a simple website. People need to see the cars and have an easy way to call and rent them. Simple enough. So what do I do? I don't fire up VS Code. I take this info straight to ChatGPT and ask it for ideas. Prompt: You're a website designer and you have a client that owns a car rental website. They want a simple website that displays the cars they have for rent and an option for people to rent them. How would you go about building this? You can see how easy that was. So what did it spit out? Basically a full project brief. It gave me a roadmap suggesting key pages like a Homepage, Car Listings, and a Contact page. It also outlined essential features like a search bar and filtering options, and recommended a modern tech stack like React which was exactly what I was planning to use. With that sorted, I wanted to see what it might look like, so I had it generate some quick wireframes. Prompt: From the above, Generate the wireframes of what the entire website with its pages will look like. Now I've got a blueprint. The whole discovery phase, which could take hours or days of back-and-forth, is done in minutes. Okay, I've got a rough idea of the layout. Time to turn these ugly wireframes into a real design. For this, I use AI-powered UI generation tools (you can find a few out there, like https://stitch.withgoogle.com, or even use v0.dev to get ideas). I just uploaded the wireframes from ChatGPT and told it what I wanted. Prompt: Turn these wireframes into a clean, modern design for a car rental website. Make it look trustworthy. Now, one thing I love about these tools is that they don't just spit out a pretty picture. They give you the actual code for it. Here's a sample of the kind of clean HTML it gave me for a single car card: You can always play with the full code here. And just like that, I've got the design of the website and the starter code for it. No Figma, no slicing assets, just straight from an idea to code. I said earlier that AI can write better code than you, and I stand by it. It was trained on all the code from every public repo, every tutorial, every developer put together. Assuming the collective brain of every developer is better than you alone, the AI has a serious edge - if you can guide it. For my car rental site, I wanted to use React. So I just copied the HTML code from the design tool and pasted it into Gemini with some very clear instructions. Notice how I was super specific about the tools I wanted? If you want the best output, you have to tell the AI exactly what you want. Don't be vague. Guide it. This means you'll need to be familiar with and understand the tools needed to create this kind of project. Overall, it took maybe ten minutes from the time I got the message from the client to the time I had a working React app running on my machine. A website built in ten minutes or less. This was not possible a while back, but with AI helping, you can move insanely fast. Look, I know this is far from done. The AI gave me a great start, but it's not a finished product. I still have to plug in a CMS or a database, set up the real logic - you get the idea. This is where the real development starts, and AI is still my co-pilot. The AI did a surprisingly good job on the first pass. It correctly scaffolded the Vite + React + TS project, created a folder, and even used components where I asked. This saved me at least 30-45 minutes of tedious setup. But it wasn't perfect. For example, the initial data for the cars was hardcoded directly inside the component. That's a huge no-no for a real app that needs to scale or pull from a database. Also, the components weren't as reusable as I'd like. This is where your job as a developer comes in - to review, refactor, and architect properly. I constantly go back to the AI to refine the code. I treat it like a pair programmer. Here's an example. The AI first gave me a component that looked something like this: Before Refactoring (AI's First Draft): This is fine for a demo, but useless for a real application. So, I guided the AI to refactor it. I'd ask it something like, "Refactor this component to accept props for car data (name, price, image) and a function for the rent button click." After Refactoring (My Guided Version): See the difference? Now it's a reusable, type-safe component that gets its data from outside. It's a back-and-forth conversation. I write some code, the AI cleans it up. The AI writes some code, I fix the logic. It's pair programming on steroids. The game has changed. AI is a tool, probably the most powerful one we've ever been given. It automates the boring stuff so we can focus on the hard problems - architecture, performance, and user experience. The developers who ignore this are going to be lapped by the ones who embrace it. It's about working smarter, not harder. Q: What's the best AI model to use? ChatGPT or Gemini or something else? A: Honestly, they're all great at writing code, and it's all a matter of "Garbage in, Garbage out." The results you get are only as good as your prompts. But if I had to choose one right now specifically for writing and refactoring code, I'd probably pick Gemini. Your mileage may vary. Q: Will I forget how to code if I rely on AI? A: That's on you. If you just copy and paste without understanding what's happening, then yeah, your skills will get dull. But if you use it to learn, to see different ways of solving a problem, and to check your own work, it'll actually make you a much better developer, faster. Q: Is it ethical to use AI for client work? A: Of course. Your client is paying you for a working website, not for your blood, sweat, and tears typing every single bracket. Is it unethical to use a framework like React or pull in a package from npm? No. This is the same thing. It's a tool. Just make sure the final product is solid, because you're the one who is ultimately responsible for it. Q: What about bugs? Does AI write perfect code? A: Heck no. It will give you buggy code. It will make things up. Don't trust it blindly. My rule is to treat code from an AI like it came from a talented but very eccentric junior dev. You have to check their work. Run it, test it, and if it breaks, you can even paste the buggy code back into the AI and say, "Hey, fix this." It's surprisingly good at cleaning up its own mess. If you have any questions, feel free to find me on Twitter at @sprucekhalifa, and don't forget to follow me for more tips and updates. Happy coding!