OpenAI Contractors Read ChatGPT Chats Under Project Lily, Raising Privacy Concerns

OpenAI's Hidden Human Review System

OpenAI employs hundreds of human contractors to review real ChatGPT conversations through an internal initiative called Project Lily, according to leaked documents obtained by 404 Media1

1

. These contractors, known as "prompt reviewers," analyze anonymized chat logs to assess response quality and guide AI training, earning over $50 an hour for what they describe as "very rote" work1

1

4

4

. The revelation challenges assumptions about how AI chatbots improve and exposes a significant gap between user expectations and actual data handling practices.

How Humans Reading ChatGPT Prompts Improve AI Responses

Human contractors review chats to eliminate specific problematic behaviors in ChatGPT responses. They evaluate four potential AI responses on a 1-to-7 scale, specifically targeting "AI-speak," unnecessary emojis, and sycophancy3

3

. Anthropomorphizing remains strictly prohibited—ChatGPT cannot claim human experiences like "as a chef, I like to..." though it can state "I found some information"1

1

. Training documents instruct reviewers to flag answers where the bot excessively flatters users or validates irrational thoughts, addressing sycophancy that has drawn legal scrutiny over user safety3

3

. Contractors describe frequently changing guidelines that sometimes contradict previous instructions, making the work "all over the place"2

2

.

Sensitive Personal Information Slips Through Anonymization

While OpenAI strips usernames and runs conversations through an automated "Privacy Filter," the system has major vulnerabilities3

3

. OpenAI openly admits the filter can make mistakes, missing uncommon identifiers or ambiguous personal context1

1

. More concerning, 404 Media discovered that contractor dashboards often include a "user memories summary" providing overviews of previous interactions that can inadvertently reveal general location, profession, or personal life context3

3

4

4

. Some reviewed prompts showed users explicitly asking ChatGPT to "keep this between us," unaware that prompt instructions don't override backend data collection4

4

. When asked if users knew about this practice, one contractor replied: "No. I don't think they would imagine some contractor somewhere is analyzing the conversations"4

4

.

User Privacy Concerns and False Sense of Intimacy

The practice raises significant user privacy concerns, especially for the 900 million active users who rely on ChatGPT as a confidant, therapist, or advisor5

5

. Michal Luria, a senior research fellow at the Center for Democracy and Technology, noted that "chatbot interfaces automatically create a false sense of intimacy and privacy"4

4

. This differs fundamentally from social media content moderation, where publishing already carries expectations of platform oversight and public exposure4

4

. The clean, empty chatbot window creates an illusion of private conversation that doesn't match the reality of data handling practices behind the scenes.

How to Opt Out of Data Sharing

OpenAI enables the "Improve the model for everyone" setting by default on Free, Plus, and Pro plans, requiring users to actively opt out of data sharing2

2

. Enterprise, Business, and Educational accounts have this disabled automatically1

1

. To stop future conversations from reaching contractors: click your profile icon, select Settings, navigate to Data Controls, and turn off "Improve the model for everyone"2

2

. However, the opt-out isn't retroactive—it only protects new conversations moving forward, while prompts already de-identified and queued remain part of the AI training cycle3

3

. Using temporary chats ensures logs won't appear in chat history, won't create memories, and won't be used in model training2

2

.

OpenAI's Shifting Transparency on Privacy Policy

OpenAI initially provided no answer to 404 Media's inquiry about whether users were explicitly informed that humans might read their chats1

1

. The company eventually pointed to an FAQ page discussing human review for model improvement, verified to be at least two years old1

1

. After 404 Media published its exposé, OpenAI changed its help page explaining opt-out procedures, though it still contains no mention of human operators1

1

. This reactive approach to transparency suggests the company may be conscious about disclosure gaps in its privacy policy.

Industry-Wide Data Handling Practices

This data collection and review pattern extends across AI chatbots industry-wide. Google Gemini clearly states in its Privacy Hub that "humans may review some saved chats"1

1

. Anthropic maintains a similar stance with a dedicated page on this topic, confirming it uses human reviewers for Claude when users leave training enabled3

3

. Perplexity's privacy policy remains unclear, neither confirming nor denying human access to chat logs1

1

. The existence of Project Lily reveals that contrary to the image these companies cultivate, models don't improve solely through technological advancement and better training sets—competent humans remain essential to the AI training process1

1

5

5

. The golden rule remains simple: if you wouldn't feel comfortable having a stranger read it, don't type it into a chatbot prompt3

3

.