Microsoft Sponsors $10,000 Challenge to Hack LLM-Integrated Email Service

Microsoft offers $10K for hackers to hijack LLM mail service

Microsoft and friends have challenged AI hackers to break a simulated LLM-integrated email client with a prompt injection attack - and the winning teams will share a $10,000 prize pool. Sponsored by Microsoft, the Institute of Science and Technology Australia, and ETH Zurich, the LLMail-Inject challenge sets up a "realistic" (but not a real, says Microsoft) LLM email service. This simulated service uses a large language model to process an email user's requests and generate responses, and it can also generate an API call to send an email on behalf of the user. As part of the challenge, which opens Monday, participants take on the role of an attacker sending an email to a user. The goal here is to trick the LLMail service into executing a command that the user did not intend, thus leaking data or performing some other malicious deed that it should not. The attacker can write whatever they want in the text of the email, but they can't see the model's output. After receiving the email, the user then interacts with the LLMail service, reading the message, asking questions of the LLM (i.e. "update me on Project X"), or instructing it to summarize all emails pertaining to the topic. This prompts the service to retrieve relevant emails from a fake database. The service comes equipped with several prompt injection defenses, and the attacker's goal is to bypass these and craft a creative prompt that will trick the model into doing or revealing things it is not trained to. Both of these have become serious, real-life threats as organizations and developers build applications, AI assistants and chatbots, and other services on top of LLMs, allowing the models to interact directly with users' computers, summarize Slack chats, or screen job seekers before HR reviews their resumes, among all the other tasks that AIs are being trained to perform. Microsoft has first-hand experience with what can go wrong should data thieves hijack an AI-based chatbot. Earlier this year, Redmond fixed a series of flaws in Copilot that allowed attackers to steal users' emails and other personal data by chaining together a series of LLM-specific attacks, beginning with prompt injection. Author and red teamer Johann Rehberger, who disclosed these holes to Microsoft in January, had previously warned Redmond that Copilot was vulnerable to zero-click image rendering. Some of the defenses built into the LLMail-Inject challenge's simulated email service include: Plus, there's a variant in the challenge that stacks any or all of these defenses on top of each other, thus requiring the attacker to bypass all of them with a single prompt. To participate, sign into the official challenge website using a GitHub account, and create a team (ranging from one to five members). The contest opens at 1100 UTC on December 9 and ends at 1159 UTC on January 20. The sponsors will display a live scoreboard plus scoring details, and award $4,000 for the top team, $3,000 for second place, $2,000 for third, and $1,000 for the fourth-place team. ®

Microsoft challenges you to hack its LLM email service

Software developers and hackers often work together to discover and fix flaws Are you an experienced hacker looking to make a little extra money this Christmas? Well you might be in luck, as Microsoft is sponsoring a competition, alongside the Institute of Science, and Technology Australia, and ETH Zurich, in which contestants will try to break a simulated Large Language Model (LLM) integrated email client. Winning teams for the LLMail-Inject challenge will be awarded a share of the $10,000 prize pool. Participants will need to sign into the challenge using a GitHub account, and create a team. The teams will then be asked to evade prompt injection defenses in a simulated LLM-integrated email client. The LLmail service includes an assistant which can answer questions and perform actions on behalf of the user, and crucially includes defenses against indirect prompt injection tasks. By bypassing the injection defenses, the hackers will be looking to prompt the LLM to do or reveal things it is not trained to. Through this, Microsoft is aiming to identify weaknesses in its current prompt injection defenses, and encourage the development of robust security measures. The relationship between security researchers and software developers is often used this way, with Google often offering a 'bug bounty' for anyone who discovers and is able to exploit vulnerabilities in its Google Cloud Platform. Similarly, Microsoft recently announced it was hosting its own Black Hat-esque hacking event, in which competitors would look for vulnerabilities in Microsoft AI, Azure, Identity, Dynamics 365, and M365. Taking a proactive approach to addressing potential vulnerabilities allows software companies to mitigate the risks before they can be exploited by threat actors in real world scenarios. Slack's AI assistant was on the receiving end of malicious prompt injections, which was luckily discovered by security researchers - but could have led to real security concerns.