Gboard's Sign-to-Text feature uses AI to translate sign language into text via camera

Gboard could soon be able to understand sign language with your phone's camera

Privacy protections would keep your video on device, and only share raw gesture data with Google's cloud AI. Google's Gboard keyboard wants nothing more than to help you get the thoughts in your head turned into words on your screen, as easily as possible. We already have so many different ways to accomplish that, from good old-fashioned tapping away at letters, to the oh-so-slick "Glide" swipe input, to the modern convenience of voice typing. But now Google could be about to seriously level up its accessibility game, as Gboard gets ready to understand sign language. Being able to automatically convert sign language gestures into written text sounds almost fantastical, but with improvements in machine vision and AI processing, the stars are finally aligning to make this feat not just possible, but practical. Last year, Google Deep Mind teased its advanced SignGemma model for interpreting sign language input, and now we may finally be getting a look at one of its first major uses. Checking out the changes present in version 17.8.3.939743344-beta-arm64-v8a of the Gboard app, we're able to get a very early look at a new Sign-to-Text input option: While we're not yet able to get the tool actually working, this introductory pop-up gives us an early overview of what to expect. As you can see, Google's building this with a hybrid local/cloud model, processing your actual video input on-device in order to extract the raw gestures, and then sending those to Google for analysis and extraction of the words you're trying to communicate. While that sounds a bit cumbersome, there are clear privacy benefits to not sharing the actual video of you signing, if that's not totally necessary. Despite Google doing that last leg of actual processing in the cloud, we are very curious if there will be any device restrictions on precisely which phones will be able to take advantage of Sign-to-Text. Digging around in the app for text strings related to this capability, we find messages meant to help users improve their visibility to the camera: Even with a strong general sense for how Sign-to-Text will likely operate, we still have plenty of questions about the all-important details here. For instance, it feels probable that Google would support American Sign Language right out of the gate, but what about British Sign Language, or any of the dozens of other regional variants around the world? Right now, we don't yet have any real insight into how those options could line up. While there's clearly still more to learn about this tool, it's already one that's easy to get excited about. This sort of input solution is exactly the sort of thing smartphones are uniquely equipped to deliver, and the fact that this one could open up new communication options for users who need them most is just really cool to see Google working on. Hopefully we're able to give you a preview of it in operation soon.

Gboard's next big feature could translate your sign language into text

Timi is the news and deals reporter for Android Police, who has been reporting on technology since 2008. He has worked in tech retail and also the IT space, providing hardware and software support, which gives him a unique perspective on the tech that he covers. This allows him to effectively break down complex subjects into easy-to-read pieces that even casual readers can enjoy. Before joining Android Police, he was a news writer for XDA, where he eventually transitioned to covering deals. He also worked as an editor and reporter for Neowin, where he covered news and attended major tech events like CES. He also reviewed phones, tablets, PC products, and other devices. In addition, he also created video content for the Neowin YouTube channel. You can type, glide, and even use your voice to compose a message. But it looks like Google isn't quite satisfied with just that, and a discovery shows that the company is working on bringing sign language on board as a new way to communicate on your Android phone. More specifically, this feature would be a part of Gboard, which would be able to read sign language and convert it into text. This one if going to be huge Android Authority uncovered Gboard's Sign-to-text feature by digging through the code of the beta version of the app. While the news outlet wasn't able to get the feature working, it was able to provide some images of the interface, giving us a small peek of what's to come. When activated, Sign-to-text will let you know that you can just sign to the phone, and it will convert it to text using the power of AI. Google adds a disclaimer, sharing that the video never goes to the cloud, but the data points for the sign language will for processing purposes. These types of advancements can be huge, so we always love to see more accessibility features from Google since they are extremely important. Will it roll out support for all languages? Now, while all of this is exciting, there is the question of what languages it will be supporting. If you aren't aware, many different types of sign language are used around the world. Because of this, there's a good chance that it may only launch in specific regions at first. On the opposite end, while it's highly unlikely, if Google has training data, it may be able to launch this feature globally. Again, this is just something in the code for now, so there's a chance that we have a long road ahead of us to see it launch to the public app. While it's exciting to be able to see what Google is working on, we always have to be ready to be disappointed because there's always the chance that Google could shelve this feature too. For now, we can only wait and see, and hope that there will be another update that signals it's closer to a public release. Regardless of what happens, Gboard will be the standard for most people. Not only does it provide an accurate typing experience, but it also offers features that cannot be found on any other third-party keyboard. Many folks experiment but find that Gboard's offerings are just far superior when compared to its competitors.