AI Hallucination Nearly Sparked Military Confrontation as US Prepared to Board Chinese Ship

AI Chatbot Hallucinated Nuclear Threat, Prompting Near-Interception

The US military came dangerously close to boarding a Chinese ship in the Middle East this spring after an AI hallucination generated false intelligence about nuclear weapons components aboard the vessel1

1

2

2

. According to CNN reporting citing four sources familiar with the episode, military aircraft were already airborne and armed personnel were actively preparing to board the Chinese ship when officials discovered the intelligence report was entirely false3

3

5

5

. One source told CNN the AI-powered fiasco almost started a war with China1

1

.

The false intelligence report originated with a Special Operations Command analyst who used an AI chatbot to synthesize open-source intelligence with classified signals intelligence from government holdings1

1

2

2

. The chatbot inaccurately identified the material the Chinese ship was carrying, wrongly flagging it as components for a nuclear weapons program4

4

. Compounding the error, the analyst then used the AI tool a second time to format the erroneous findings into an official-looking summary that circulated across command channels during the conflict with Iran2

2

5

5

.

Pentagon's AI-First Push Creates New Vulnerabilities

This near-miss exposes critical vulnerabilities in the Pentagon's aggressive push to integrate AI across military operations. In January, Defense Secretary Pete Hegseth rolled out an AI acceleration strategy directing the Department of Defense to make all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component1

1

. Hegseth emphasized that AI is only as good as the data it receives, promising to ensure data availability1

1

.

The military has rapidly deployed multiple AI systems for intelligence work. Last December, the Department of Defense announced it would use Gemini for Government as the basis for its bespoke GenAI.mil platform1

1

. The following month, Grok for Government was added as an option on the platform1

1

. Anthropic also offers a customized version of Claude for US spy work1

1

. In June, a Pentagon representative revealed to Congress that 1.5 million active DoD personnel have used the military's generative AI tools, and that the department uses generative AI to help create congressionally mandated reports1

1

.

Speed Versus Safety in Military Decision-Making

The Pentagon has described AI in military operations as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time2

2

. However, the same speed that makes AI attractive in military decision-making may also allow hallucinations to propagate through command structures with insufficient human oversight2

2

.

Jake Steckler, research scholar at GovAI and veteran US Army officer, emphasized the stakes: It's important for service members to understand the uncertainty inherent to LLMs, but it's especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions2

2

. Steckler warned that prioritizing adoption speed over safeguards will likely lead to incidents that only make service members lose trust in these systems, which ultimately will slow adoption2

2

.

As one military source told CNN, AI in targeting is definitely something that is ramping up, and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide5

5

. This comment raises questions about whether current protocols adequately address the risks of AI integration in sensitive military contexts.

Broader Pattern of AI-Driven Accidents

This incident represents one of the more potent and consequential instances of a hallucinating AI ruining the reliability of a professional report1

1

. Since hallucinating became the Cambridge Dictionary's word of the year in 2023, prominent examples have emerged of non-fiction authors, journalists, academic researchers, judges, doctors, police departments, corporate call centers, and more getting taken in by AI tools that simply make something up when their training data doesn't provide sufficient context1

1

. Despite attempts at do not hallucinate prompts, some researchers suggest it may be impossible to prevent LLMs from hallucinating altogether1

1

.

A 2023 State Department Declaration on Responsible Military Use of Artificial Intelligence and Autonomy stressed that principled use of AI by armed forces should include careful consideration of risks and benefits, minimize unintended bias and accidents, and maintain accountable use with a human in the loop and a responsible human chain of command and control1

1

. Yet the gap between policy statements and operational reality appears significant.

What This Means for Military AI Adoption

The incident calls into question the rapid pace with which AI has been integrated into every corner of the US government, from transit regulation to the Pentagon and every office in between5

5

. Such tools have already been used in the bombing of Iran and ongoing air strikes on fishing boats in the Caribbean5

5

.

Watch for increased scrutiny of AI partnerships between tech companies and the military. In March, the Department of Defense blacklisted Anthropic over the company's opposition to its models' use in autonomous weapons systems, a move a federal judge said last month was unlawful retaliation in violation of the First Amendment1

1

. SpaceXAI reportedly struck a deal to let the military use Grok in February, while NVIDIA, Microsoft and Amazon entered into a partnership to offer their AI technologies to the DoD in May4

4

. These government deals are lucrative, which is why companies pursue them, but helping spread AI adoption clearly carries risks that normal enterprise software doesn't4

4

.

The reported near-miss comes as extinction-level warnings from AI researchers have led to a newly prominent national conversation on AI safety, including calls for regulation and coordinated research pacing from leading frontier AI labs1

1

. Expect demands for stronger safeguards, verification protocols, and ethical concerns to intensify as more AI-driven accidents surface in high-stakes military contexts.