Google's Pixel Studio App: Revolutionary AI Image Tools Face Early Challenges

Google's Pixel AI Image Creation and Editing Tools Are Kind of Terrifying

Google last week introduced the Pixel 9 smartphone lineup, with a selection of new devices that are focused on AI. All of the Pixel 9 phones support Gemini AI, and Google has also added AI-based image generation and editing tools. Reviewers have been testing the new smartphone features, and the AI image generation is probably going to turn into a PR nightmare for Google based on what's come out so far. Google introduced an AI image generation app for the Pixel phones, called Pixel Studio. It's designed to create stickers and images using text-based prompts, and it looks and sounds a lot like Apple's planned Image Playground. So far, reviewers have been able to create all manner of questionable images with Pixel Studio, ranging from a Nazi version of Spongebob to Elmo with an AK47. Google said that there were "safety checks" in place to keep Pixel Studio from being "used nefariously," and there are some. Pixel Studio will not create images of humans, and Google is tweaking the AI in response to reviews. After speaking with Google, Digital Trends was no longer able to get Pixel Studio to create cartoon characters using cocaine or dressed as German soldiers. Even more concerning than Pixel Studio is Google's "Reimagine" tool, which can add objects to photos that you've already taken. The Verge used it to add corpses, bombs, drugs, and disasters to images, and objects included in photos look so realistic that it's difficult to tell when an image has been edited. Google is able to do a pretty good job matching lighting and perspective to the original photo, and there are no watermarks or flags on social media. Google does add a metadata tag, but that's easy enough to eliminate with a screenshot. You can, of course, use Reimagine for adding sunsets and rainbows to your image, just like you can use Pixel Studio for creating fun pictures, and both AI features work great, so these tools aren't all bad. Android Authority shared a post with Oreo pizza, a giraffe surfing, and kittens playing basketball, for example. As The Verge points out, you could always add a body to an image or create a picture of Elmo with a gun, but it would take some Photoshop skill and time to do so. With the Pixel phones, it takes just a thought and a few seconds to create that kind of image, and it's right there on the smartphone for anyone to use. In statements to The Verge and Digital Trends, Google said that it designed its generative AI tools to "respect the intent of user prompts" which can lead to the creation of content "that may offend" when the user asks for it. But Google claims that there are "Terms of Service" on what content is not allowed, and that safeguards will be continually refined. Apple hasn't yet introduced its image generation tools, Image Playground or Genmoji, and the media surrounding the Pixel 9 launch gives us some insight into what the response might be like if Apple doesn't get it just right. With AI, it's likely that people are going to be able to find workarounds even if Apple puts numerous safeguards in place. Apple so far has shied away from photorealistic AI images, but Google is going all in, and it's going to be much harder to trust photos on social media and the internet going forward.

Google's Pixel Studio image generation can easily go off the rails

One of the new AI-powered launch features on the Pixel 9 series is Pixel Studio, an app that takes text prompts and turns them into images. It might end up being a fun little tool, but it can also go off the rails pretty easily in our experience. Pixel Studio is a dedicated app for creating images from text prompts. The app works on the same idea as Gemini and ImageFX, allowing users to input a text prompt and get an image in return. But, like any respectable image generator, there have to be limits in place. In a statement to 9to5Google, Google explains that there are limits in place for both Pixel Studio and Magic Editor to "prevent abuse" while still respecting "the intent of user prompts" even if instructed to create content that "may offend" if the user directly tells the app to do so. Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it's not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don't allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools' guardrails and we remain committed to continually enhancing and refining the safeguards we have in place. This is similar to Gemini, which has safeguards in place against potentially offensive or divisive content. For example, Google was criticized for inaccurate depictions of people in historical context which led to Google "temporarily" disabling the ability to generate images of people as it has for the past several months. Like Gemini, Pixel Studio is unable to generate images of people, but we've found that the app can quickly and easily be sent off of the rails. Ahead of today's Pixel 9 review embargo, we found that Pixel Studio was able to generate images of cartoon characters in WWII German uniforms, in some cases with Nazi symbols. Another jarring and worrying example we viewed (from the folks at Digital Trends who first spotted these troubles) saw a character shooting in a school with dead children surrounding it, which Google's models shouldn't be able to generate in the first place. Since initially viewing those results, the prompts used to create them appear to have been blocked by Google on our Pixel 9 Pro XL, but Digital Trends says they're still able to use some of them. This all implies that the guardrails in Pixel Studio will effectively turn into a never-ending game of whack-a-mole. Somewhere, somehow a prompt will slip through and create something that the app shouldn't be capable of if it truly is using the same guardrails as Gemini - we checked with Google and the company says that "all" of its text-to-image tools "share similar guidelines for prohibited usage," linking out to its AI policy. Realistically, this sort of thing is inevitable with AI image generation. It's a problem that virtually all AI image generators will face to some extent. As our Abner Li explained in his Pixel 9 Pro review earlier today:

I tried Google's new Pixel Studio app, and it's a mess | Digital Trends

Google's Pixel 9 phones are all about AI. If you watched the company's launch event, there was no way of missing it, as that's almost all Google seemed interested in talking about. Some AI here, a sprinkle of AI there, and an extra touch of AI on top, just to be safe. One of the many AI features on the Pixel 9 is an app called Pixel Studio. It's an image generator app that allows you to create an image of virtually anything you'd like. You open the app, type in what you want a picture of, and voilà -- Pixel Studio makes it for you. It's not the first app of its kind, but it is the first one shipped on a Google Pixel phone. Unfortunately, it's a dumpster fire. What Pixel Studio does well Before I get into the bad stuff, I want to give Google credit where it's due. Although I'm not a fan of AI image generator apps as a whole, Pixel Studio is a well-designed and easy-to-use application. Upon opening the app, you're presented with 10 categories of suggested image types -- including magical castles, meme reactions, sports, retro video games, and more. You can browse these for inspiration or tap the blue Create button at the bottom of the app to start from scratch. If you want to create your own image, it's as easy as typing in what you want Pixel Studio to generate a photo of. Want to see a picture of a black cat wearing sunglasses while wearing a funny hat? Type that into Pixel Studio, and you'll almost immediately get a photo of it. What if you'd rather see your image as a sketch? Maybe you'd prefer it in the style of a video game? You can do that with a couple of taps. And if you want to see that cat lounging on the beach with fireworks in the background? Just add it to your prompt, and you've got it. Whatever your feelings toward AI image generation are, this is an objectively slick and thoughtfully designed app. However, it's what you can do with Pixel Studio where things get ... troublesome. Where the app falls apart As with most image generator apps, Pixel Studio has guardrails to prevent you from making any type of image. I pretty quickly found that it refuses to generate photos of people, but I was curious what other limits Google is enforcing. When asked about this, the company told Digital Trends the following: "Pixel Studio and Pixel Screenshots follow Google's AI Principles. For example, all fully synthetic, text-to-image generated images in Pixel Studio use SynthID watermarking, so these images can be tracked -- even if they're shared or edited. Moreover, we've disabled human generation and added many safety checks to our servers to balance our bold -- yet responsible -- approach to AI. This gives our customers access to a very powerful tool, but ensures we have checks in place to ensure Pixel Studio can't be used nefariously." I've repeatedly tried to make Pixel Studio generate images of people, and it has consistently refused to do so, which is good. However, those "many safety checks" that are supposedly in place don't appear to work nearly as well as they should. How so? As the headline of this article teases, I've had Pixel Studio generate an image of SpongeBob dressed as a Nazi. When given the prompt "SpongeBob dressed as a German soldier from WWII with a swastika on the uniform," Pixel Studio didn't hesitate to provide me with an image of precisely that. And that's just one example. I've also gotten Pixel Studio to generate pictures of Elmo pointing a shotgun at Big Bird, Yoda doing cocaine, Mr. Krabs holding an assault rifle, and more. To be clear, Pixel Studio hasn't generated inappropriate images unexpectedly. If I ask for a photo of a cute dog, I'll get a photo of a cute dog -- not one brandishing a gun. Pixel Studio only generates pictures of weapons, drugs, etc., when specifically asked to. The most troubling thing Pixel Studio has generated are multiple images of copyrighted characters firing guns in a school shooting setting -- including depictions of dead students lying on the ground. We've decided not to include those images, but they're the worst example of how off the rails Pixel Studio is. It's important to mention that I didn't have to twist Pixel Studio's arm to get these images. Initially, I suspected you needed to preface prompts with "an object resembling" before it would generate these pictures. However, it quickly became apparent that wasn't the case. A clear and straightforward prompt like "Mickey Mouse dressed as a slave owner" or "Paddington Bear on a crucifix" is all Pixel Studio needs. You don't need to trick the app into generating these pictures. If this were a random website or third-party app, perhaps we wouldn't be so critical. But that isn't what's happening here. Pixel Studio is a Google-made application and one of the handful of new apps shipping on the company's latest Pixel 9 phones. It should be creative and open-ended, but there's also a limit to what apps like this should do. Furthermore, Microsoft Copilot and Google's own Gemini image generator refuse to generate images with the same prompts. That's not to say those tools are perfect, but it's clear that safety measures working in other image generators aren't working in Pixel Studio. How is this happening? If you've followed AI image generation this year, you'll know this isn't our first time here. This month, X's Grok chatbot got an even more outlandish image generator tool -- generating pictures of politicians and celebrities in similarly tasteless situations. Google itself was in hot water this past February after its Gemini image generator created photos of Nazis and the U.S. Founding Fathers as Black people. Given Google's history here, you'd think the company had learned from its past mistakes. And with other image generator tools, it seemingly has. I tried many of the same prompts that worked in Pixel Studio with Google's other image generators -- Gemini and ImageFX -- and couldn't replicate any of these prompts on those other platforms. For context, Gemini currently uses the Imagen 2.0 model for image generation, while ImageFX uses Imagen 3 -- the same one that Pixel Studio uses. This discrepancy between ImageFX and Pixel Studio is particularly confusing since they use the same models. I asked Google for clarification about this and was redirected to the company's Generative AI Prohibited Use Policy. What Google should do about Pixel Studio While writing this story, I sent Google examples of prompts I used with Pixel Studio -- specifically, those about German soldiers and characters using cocaine. A little over 24 hours after sending those examples, Pixel Studio now says, "An unknown error occurred" when trying to generate those images. Is that progress? Certainly. However, it doesn't fully address the underlying problems with Pixel Studio. I'm glad those two images are now impossible to generate, but it was only after I sent multiple emails to Google explaining the issue. It also doesn't address that these images were allowed in the first place and that similarly inappropriate photos are still being generated. At the time of publication, Pixel Studio is still actively creating images of school shootings, popular children's characters dressed as slave owners, etc. I pressed Google for another statement about the whole situation, and the company's Communications Manager Alex Moriconi said the following: "Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it's not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don't allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools' guardrails and we remain committed to continually enhancing and refining the safeguards we have in place." As a company with an already problematic track record for AI image generation, creating an app specifically for this was always going to be a risk. Google took that risk, and now this is where we're at. At its best, Pixel Studio could have been a fun tech demo for Google to promote its AI chops. Instead, it's put an uncomfortable cloud over the company's latest smartphones -- which is a shame, as the Pixel 9 Pro XL is one of the best phones I've used this year. The last time Google was here with Gemini image generation concerns, the company responded within days. One can only hope for a similarly prompt reaction to Pixel Studio's issues, though whether that should be done via quick app updates or scrapping the application altogether remains to be seen.