Google's Pixel 11 Introduces Sign Language to Text Translation for 70 Million Deaf Users

The Pixel 11 Pro Fold can turn sign language into text, here's how

The feature currently only supports ASL and is available in select countries and regions. Google today announced the Pixel 11 series, including its newest foldable: the Pixel 11 Pro Fold. While it's easy to get distracted by the foldable Pixel's sleek design and new 48MP camera, the phone also comes with a feature that could prove to be very useful. The Pixel 11 Pro Fold supports transcribing American Sign Language to text. The phone will use its camera to see and understand ASL gestures and transcribe them into text directly on the screen. This could make it easier for ASL users to communicate with others, even if they don't know sign language. Users can simply open the Translate app and tap the sign language icon in Gboard's top toolbar to start. The feature will use the Pixel 11 Pro Fold's front camera to recognize what the user is signing, and the transcription will also be displayed on the cover screen, so the other person can easily read it and respond. The feature uses Google's new sign-language-to-text (SL2T) translation model. The model can "see" and understand physical movement, including the movement of a user's hands, arms, torso, head, and face. It uses this information to understand the meaning being conveyed through the sign language and translate it into English. Google says SL2T was trained on over 100,000 hours of sign language data across more than 50 languages, with roughly a quarter of the data in ASL. The new foldable Pixel also supports transcribing speech to text, which could be a useful accessibility feature, especially for those who are hard of hearing or deaf. The feature provides a text field for typing responses or questions for the other person to read. ASL transcription is currently available on the Pixel 11 series in select countries and only supports American Sign Language-to-English translation. However, Google has confirmed that support for more devices and additional languages is coming soon. The Pixel 11 Pro Fold starts at $1,899 and is available to pre-order now. It will go on sale starting August 20 and will include a free six-month Google AI Pro subscription.

Putting sign language AI into users' hands

Introducing sign-language-to-text (SL2T), our breakthrough model powering new sign language features for Deaf and hard of hearing users. AI's ability to process spoken languages has advanced rapidly over recent decades, enabling automatic translation, dictation, and conversational interfaces that feel effortless to hearing users. Yet this technological revolution has not reached the world's more than 200 sign languages -- and the estimated 70 million Deaf and hard of hearing people who use them. Today, we're introducing a massively multilingual sign-language-to-text (SL2T) translation model that marks a breakthrough in quality and generality. With it, we are bringing sign language AI out of the lab and into consumer products for the first time: SL2T powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11, starting with American Sign Language (ASL) to English. More devices are coming soon, and additional languages will follow. Similarly to how hearing users can use dictation to speak instead of typing, this feature enables Deaf users to sign to their phone anywhere they'd normally type. You can sign to search the web, draft messages or documents, and ask Gemini to solve queries or execute tasks. In Live Transcribe, you can sign responses in conversations instead of having to type back and forth. According to our testers, signing in ASL is faster, more natural, and more delightful than typing in English.

Google's Pixel 11 can translate American Sign Language into text

Talking to your phone has long been a convenient alternative to typing. With the Pixel 11, Google is extending that option to people who communicate through American Sign Language (ASL). Ahead of its Made by Google event Wednesday evening, Google DeepMind announced SL2T, a new AI model that translates sign language directly into written text. The model powers a sign-to-text feature launching in Gboard and Live Transcribe on the Pixel 11 family. In practice, the feature allows users to use sign language via their phone's camera for tasks they would typically use keys to perform. That could mean signing a Google search, drafting a text or email, writing in a document, or asking Gemini a question. In Live Transcribe, a user can sign a response during an in-person conversation instead of typing it out. The feature initially translates ASL into English. Google says it will come to more devices soon, at no additional cost, with support for additional sign languages planned for the future. How does sign-to-text work? Teaching a phone to understand sign language takes more than showing it a dictionary of hand gestures. ASL is its own language, with distinct grammar and meaning conveyed through movements of the hands, arms, torso, head, and face, sometimes all at once. That means SL2T has to pay attention to the signer's entire upper body. An on-device computer vision system tracks 130 points across the person's face, body, and hands, essentially turning the video into a moving map of geometric coordinates. The raw footage is immediately discarded, while those coordinates are sent to Google's servers for translation, according to the DeepMind sign language team. Google's joint impact report adds that the company does not retain logs of what users sign or the text the system produces unless someone explicitly agrees to participate in a model evaluation study. Once the phone has mapped those movements, SL2T then translates them directly into English. A common approach in sign-language translation has been to first convert signs into written labels known as "glosses" before translating them into text. The problem with that prior approach, according to the blog post, is that "glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions. Translating directly from landmarks removes artificial vocabulary limits." In other words, skipping that word-by-word step allows SL2T to account for facial expressions, physical positioning, and other movements that carry meaning. Google trained SL2T on more than 100,000 hours of data spanning over 50 sign languages, with roughly one-quarter of the material in ASL. It also trained SL2T to understand one-handed signing, which is useful when someone is holding their phone with the other hand, as well as signing by left-handed users. Still, Google says SL2T is not a replacement for an interpreter The first version still has significant limitations. DeepMind's report says the model can struggle with regional signs, slang, complex ASL grammar, rapid fingerspelling, and meanings conveyed primarily through facial expressions or head movements. It may also generate what Google calls "ghost text," or words the user did not sign, when someone pauses, or another person enters the camera frame. Its accuracy can also decline in low light, when a signer is partially outside the frame, or when there is not enough contrast between a person's hands, face, clothing, and surroundings. The model has also not been formally evaluated on people under 18 or extensively tested across signers with motor disabilities. For those reasons, Google and its AI Sign Language Advisory Committee describe SL2T as "strictly an assistive input tool for low-stakes communication" and does not fulfill legal requirements for reasonable accommodations. It is not designed for medical appointments, legal proceedings, classrooms, job interviews, government hearings, or other settings where an error could carry serious consequences. The report also warns institutions against using the software to replace qualified human interpreters or avoid their accessibility obligations. Google enlisted Deaf employees, outside testers, advocacy organizations, professional interpreters, and sign language experts to guide the model's development. The company plans to continue working with its advisory committee as it expands the technology to additional sign languages and, eventually, feature AI that can generate sign language as well. SL2T is one of several new features arriving alongside Google's latest hardware. The company's Made by Google event begins Wednesday at 6 p.m. ET, with Trevor Noah hosting a presentation featuring the Pixel 11 lineup, Pixel Watch 5, and more of Google's latest product announcements. Follow along with Mashable's Made by Google live blog for updates and analysis throughout the event!

Pixel 11 introduces a new way for deaf users to communicate with people who don't know sign language

The Pixel 11's camera-based sign-to-text feature converts sign language into text for more natural face-to-face conversations. Google's new Pixel 11 series devices bring more than just hardware upgrades. One of the lineup's most interesting new features is designed to break down the communication barrier between deaf and hard-of-hearing users and those who don't know sign language. Pixel Camera can now turn sign language into text Google has introduced a new sign-to-text accessibility feature that uses the Pixel Camera to recognize sign language and automatically convert it into text, so users can sign as they normally would instead of stopping to type a response. Google says the goal is to turn the phone from a one-way listening tool into one that supports natural, two-way conversation during spontaneous, face-to-face interactions. The feature runs on a new AI model called sign-language-to-text (SL2T), built by Google DeepMind, which the company says marks a breakthrough in translating sign language at scale. Once the Pixel Camera captures a user signing, the model translates it, and Gboard converts it into text anywhere someone would normally type, including web searches, messages, documents, and Gemini queries. Recommended Videos At launch, the feature supports American Sign Language to English translations on the Pixel 11 series, with Google saying more languages and support for additional devices will follow. Built with privacy and real-world signing in mind To protect privacy, the system tracks the location points on a signer's body on-device instead of analyzing raw video, sending only those coordinates to Google's servers for translation before discarding the original camera feed. Google DeepMind says the model was trained on more than 100,000 hours of data across more than 50 sign languages, and that it specifically tested performance for left-handed signers, who make up about 10 percent of users, and for one-handed signing, common when someone is holding the phone in their other hand. The company says the model scored higher on a standard sign language translation benchmark than any previous system, though it hasn't published data on how it performs in less controlled settings, poor lighting, or fast, casual conversation. Translating sign language is a fundamentally different problem than translating speech, since sign languages have their own independent grammar and vocabulary instead of word-for-word mapping. An estimated 70 million deaf and hard-of-hearing people worldwide use sign language, and Google DeepMind says this marks the first time sign language AI at this scale has reached a consumer product. Whether SL2T holds up in fast, everyday signing, rather than the benchmark conditions Google has published so far, will determine if it becomes a genuine communication tool or another AI feature that undersells its own demo.

Google debuts SL2T, an AI model that's designed to understand sign language

Google DeepMind said today it wants to bring the artificial intelligence revolution to the estimated 70 million people across the world who are either deaf or hard of hearing with the launch of sign-language-to-text or SL2T. In a blog post, Google's AI researchers said SL2T is a multilingual translation model that's making its debut on the Pixel 11 smartphone, where it powers a new "sign-to-text" dictation feature on Gboard and Live Transcribe. The model is initially capable of translating American Sign Language or ASL to English text, and is the first of its kind to be made available within a real-world consumer product, the company said. AI's ability to process human speech has progressed enormously in the last few years, to the point where anyone can dictate anything they like in dozens of global languages on a smartphone, tablet or personal computer. But the same isn't true for those who rely on one of the more than 200 distinct sign languages because of their hearing disabilities. According to Google, it's an audience that has been completely ignored by the AI industry, until now. The SL2T model gives deaf and hard of hearing users the ability to interact with their smartphone using their native language. Just like speech AI makes it possible for users to talk to their device instead of typing, SL2T makes it possible for people to sign directly into their smartphone's camera rather than tap away entering text. It means users can now use sign language to perform dozens of different tasks, including web searches, drafting emails and text messages, editing documents and so on. They can also use SL2T to prompt Gemini to answer queries and perform actions on their behalf. The model is also available in Google's Live Transcribe app, allowing users to sign their responses directly during face-to-face calls. Google said SL2T was trained on more than 100,000 hours of sign language data spanning over 50 languages, with about a quarter of that dataset made up of ASL communications. While the initial release can only understand ASL, Google said it decided to train the model across multiple sign languages in order to learn the shared structural patterns across them. In this way, it can significantly outperform earlier sign language models. To ease users' privacy concerns, SL2T is powered by an on-device computer vision model called MediaPipe Holistic, which tracks the geometric pose locations across signer's faces, hands, arms and torso. It then sends the coordinates of these locations to its cloud-based server, avoiding the need to upload any actual video, making it faster and more secure for users. The model translates sequences directly into text, bypassing the intermediate text annotations known as "glosses." By doing this, SL2T is better able to capture the non-manual expressions and spatial grammar structures that are characteristic of ASL, Google said. Other features include optimizations to reduce latency, hallucination prevention mechanisms for non-signing movements and support for both left-handed signers and one-handed signing, so users can interact with it while holding their smartphone in one hand. Google's performance claims are backed by solid data. SL2T achieved a score of 70 BLEURT on the FLEURS-ASL benchmark. The release of SL2T is a key development for deaf communities globally, Google said. The development of AI models that can understand sign languages has been slow due to the unique challenges they present and also some common misconceptions. Unlike spoken languages, which map sequential sounds directly to text, sign languages have their own unique lexicons and grammar that must be translated in an entirely new way. Another difficulty is that sign languages convey meaning through simultaneous movements of not just the hands, but also the head, face, arms and torso. But early attempts to develop sign languages focused on hand gestures only, rather than treating them as full-body visual languages. Google signaled to the deaf and hard-of-hearing communities that it's not going to leave them behind any longer. In addition to releasing SL2T, it has also established the AI Sign Language Advisory Committee in partnership with a number of deaf organizations and sign language experts to help guide the responsible deployment of the technology. The committee will also co-author a report alongside Google that outlines SL2T's capabilities and its current limitations. Going forward, Google plans to expand the model to cover additional sign languages and also develop models for sign language generation.