Google Launches SL2T: AI Model Converts American Sign Language to Text on Pixel 11

The Pixel 11 Pro Fold can turn sign language into text, here's how

The feature currently only supports ASL and is available in select countries and regions. Google today announced the Pixel 11 series, including its newest foldable: the Pixel 11 Pro Fold. While it's easy to get distracted by the foldable Pixel's sleek design and new 48MP camera, the phone also comes with a feature that could prove to be very useful. The Pixel 11 Pro Fold supports transcribing American Sign Language to text. The phone will use its camera to see and understand ASL gestures and transcribe them into text directly on the screen. This could make it easier for ASL users to communicate with others, even if they don't know sign language. Users can simply open the Translate app and tap the sign language icon in Gboard's top toolbar to start. The feature will use the Pixel 11 Pro Fold's front camera to recognize what the user is signing, and the transcription will also be displayed on the cover screen, so the other person can easily read it and respond. The feature uses Google's new sign-language-to-text (SL2T) translation model. The model can "see" and understand physical movement, including the movement of a user's hands, arms, torso, head, and face. It uses this information to understand the meaning being conveyed through the sign language and translate it into English. Google says SL2T was trained on over 100,000 hours of sign language data across more than 50 languages, with roughly a quarter of the data in ASL. The new foldable Pixel also supports transcribing speech to text, which could be a useful accessibility feature, especially for those who are hard of hearing or deaf. The feature provides a text field for typing responses or questions for the other person to read. ASL transcription is currently available on the Pixel 11 series in select countries and only supports American Sign Language-to-English translation. However, Google has confirmed that support for more devices and additional languages is coming soon. The Pixel 11 Pro Fold starts at $1,899 and is available to pre-order now. It will go on sale starting August 20 and will include a free six-month Google AI Pro subscription.

Pixel 11 introduces a new way for deaf users to communicate with people who don't know sign language

The Pixel 11's camera-based sign-to-text feature converts sign language into text for more natural face-to-face conversations. Google's new Pixel 11 series devices bring more than just hardware upgrades. One of the lineup's most interesting new features is designed to break down the communication barrier between deaf and hard-of-hearing users and those who don't know sign language. Pixel Camera can now turn sign language into text Google has introduced a new sign-to-text accessibility feature that uses the Pixel Camera to recognize sign language and automatically convert it into text, so users can sign as they normally would instead of stopping to type a response. Google says the goal is to turn the phone from a one-way listening tool into one that supports natural, two-way conversation during spontaneous, face-to-face interactions. The feature runs on a new AI model called sign-language-to-text (SL2T), built by Google DeepMind, which the company says marks a breakthrough in translating sign language at scale. Once the Pixel Camera captures a user signing, the model translates it, and Gboard converts it into text anywhere someone would normally type, including web searches, messages, documents, and Gemini queries. Recommended Videos At launch, the feature supports American Sign Language to English translations on the Pixel 11 series, with Google saying more languages and support for additional devices will follow. Built with privacy and real-world signing in mind To protect privacy, the system tracks the location points on a signer's body on-device instead of analyzing raw video, sending only those coordinates to Google's servers for translation before discarding the original camera feed. Google DeepMind says the model was trained on more than 100,000 hours of data across more than 50 sign languages, and that it specifically tested performance for left-handed signers, who make up about 10 percent of users, and for one-handed signing, common when someone is holding the phone in their other hand. The company says the model scored higher on a standard sign language translation benchmark than any previous system, though it hasn't published data on how it performs in less controlled settings, poor lighting, or fast, casual conversation. Translating sign language is a fundamentally different problem than translating speech, since sign languages have their own independent grammar and vocabulary instead of word-for-word mapping. An estimated 70 million deaf and hard-of-hearing people worldwide use sign language, and Google DeepMind says this marks the first time sign language AI at this scale has reached a consumer product. Whether SL2T holds up in fast, everyday signing, rather than the benchmark conditions Google has published so far, will determine if it becomes a genuine communication tool or another AI feature that undersells its own demo.

Google debuts SL2T, an AI model that's designed to understand sign language

Google DeepMind said today it wants to bring the artificial intelligence revolution to the estimated 70 million people across the world who are either deaf or hard of hearing with the launch of sign-language-to-text or SL2T. In a blog post, Google's AI researchers said SL2T is a multilingual translation model that's making its debut on the Pixel 11 smartphone, where it powers a new "sign-to-text" dictation feature on Gboard and Live Transcribe. The model is initially capable of translating American Sign Language or ASL to English text, and is the first of its kind to be made available within a real-world consumer product, the company said. AI's ability to process human speech has progressed enormously in the last few years, to the point where anyone can dictate anything they like in dozens of global languages on a smartphone, tablet or personal computer. But the same isn't true for those who rely on one of the more than 200 distinct sign languages because of their hearing disabilities. According to Google, it's an audience that has been completely ignored by the AI industry, until now. The SL2T model gives deaf and hard of hearing users the ability to interact with their smartphone using their native language. Just like speech AI makes it possible for users to talk to their device instead of typing, SL2T makes it possible for people to sign directly into their smartphone's camera rather than tap away entering text. It means users can now use sign language to perform dozens of different tasks, including web searches, drafting emails and text messages, editing documents and so on. They can also use SL2T to prompt Gemini to answer queries and perform actions on their behalf. The model is also available in Google's Live Transcribe app, allowing users to sign their responses directly during face-to-face calls. Google said SL2T was trained on more than 100,000 hours of sign language data spanning over 50 languages, with about a quarter of that dataset made up of ASL communications. While the initial release can only understand ASL, Google said it decided to train the model across multiple sign languages in order to learn the shared structural patterns across them. In this way, it can significantly outperform earlier sign language models. To ease users' privacy concerns, SL2T is powered by an on-device computer vision model called MediaPipe Holistic, which tracks the geometric pose locations across signer's faces, hands, arms and torso. It then sends the coordinates of these locations to its cloud-based server, avoiding the need to upload any actual video, making it faster and more secure for users. The model translates sequences directly into text, bypassing the intermediate text annotations known as "glosses." By doing this, SL2T is better able to capture the non-manual expressions and spatial grammar structures that are characteristic of ASL, Google said. Other features include optimizations to reduce latency, hallucination prevention mechanisms for non-signing movements and support for both left-handed signers and one-handed signing, so users can interact with it while holding their smartphone in one hand. Google's performance claims are backed by solid data. SL2T achieved a score of 70 BLEURT on the FLEURS-ASL benchmark. The release of SL2T is a key development for deaf communities globally, Google said. The development of AI models that can understand sign languages has been slow due to the unique challenges they present and also some common misconceptions. Unlike spoken languages, which map sequential sounds directly to text, sign languages have their own unique lexicons and grammar that must be translated in an entirely new way. Another difficulty is that sign languages convey meaning through simultaneous movements of not just the hands, but also the head, face, arms and torso. But early attempts to develop sign languages focused on hand gestures only, rather than treating them as full-body visual languages. Google signaled to the deaf and hard-of-hearing communities that it's not going to leave them behind any longer. In addition to releasing SL2T, it has also established the AI Sign Language Advisory Committee in partnership with a number of deaf organizations and sign language experts to help guide the responsible deployment of the technology. The committee will also co-author a report alongside Google that outlines SL2T's capabilities and its current limitations. Going forward, Google plans to expand the model to cover additional sign languages and also develop models for sign language generation.