Amazon Enhances Checkout-Free Technology with Advanced AI

Amazon Upgrades Checkout AI to Boost Third-Party Adoption

As Amazon looks to license its Just Walk Out checkout system to more stores, the eCommerce giant is upgrading the artificial intelligence model to improve accuracy. The company announced in a blog post Wednesday (July 31) that the new multimodal AI foundation model uses advanced machine learning techniques, akin to those used in generative AI applications, to analyze data from cameras and sensors concurrently. This marks a departure from the previous sequential data analysis method, addressing issues such as camera obstructions and complex shopper behaviors more effectively. "What we realized in the past couple of years is ... we could combine all of these different signals into a single model and ... teach a machine learning system to operate on the entirety of that information all at once to produce a receipt," Jon Jenkins, vice president of Just Walk Out technology, shared on a call. "... When we took this multimodal, single foundation model approach, we found out that we could generate receipts faster, more efficiently and more accurately than we were able to with the existing system, which was already pretty good." Jenkins explained that this foundational technology allows Just Walk Out to process and integrate multiple types of data inputs -- video from cameras, weight signals from sensors, RFID signals and images of items taken off shelves -- into a comprehensive model, which is "kind of a game changer." Many consumers, for their part, are interested in this kind of checkout system, according to the PYMNTS Intelligence study "How We Will Pay Report: How Connected Devices Enable Multitasking Among Digital-First Consumers." Specifically, among the 95% of consumers who own connected devices, 35% would be interested in an experience where they walk into a store, pick up a product and walk out without standing in a checkout line to pay. Sensors detect what they have put in their cart, and they are automatically charged for those items using an app with the payment method. Additionally, 6% said they already do this. Amazon's Just Walk Out technology is gaining traction beyond its own stores, with over 170 installations in third-party locations, Jenkins said. Initially launched under Amazon's retail division, the project shifted to AWS around two years ago to better serve non-Amazon businesses. "There are going to be way more non-Amazon stores than Amazon stores," Jenkins remarked, highlighting the need to cater to a broader market. The pivot focuses on making the technology easily installable and operable for a range of external customers. Jenkins emphasized that Amazon's own stores, like Amazon Go and Amazon Fresh, are now treated as just another customer among many. Most new installations are third-party locations, including airports and stadiums. For instance, Hudson operates 16 Just Walk Out-enabled stores in airports across the country, and several stadiums, including Lumen Field and the Washington Commanders' stadium, have embraced the technology. Jenkins highlighted several key areas where the technology is beginning to catch on. The initial large-scale third-party deployments of Just Walk Out were in stadiums across the country. Jenkins cited Lumen Field as an example, where the technology increased customer throughput by 85% and more than doubled revenue in the same space. Additionally, in cases where stores need to, say, stay open overnight, the technology, operating unattended, is meant to present a viable financial model during those off-peak hours. Plus, in areas that see short bursts of high traffic, the technology is meant to allow these venues to manage increased demand without increasing labor costs, optimizing staffing efficiency. Jenkins also stated that high-theft locations see a marked decrease in shrinkage with Just Walk Out. The technology effectively converts potential theft into legitimate purchases, as items taken off the shelf are automatically added to the customer's virtual cart. Additionally, for new facilities, Just Walk Out allows architects to allocate less space to retail areas. This reduction, Jenkins said, translates into cost savings in construction and operational efficiency.

Amazon introduces new AI model to make checkout-free 'Just Walk Out' stores even better - SiliconANGLE

Amazon introduces new AI model to make checkout-free 'Just Walk Out' stores even better Amazon.com Inc. today introduced a new artificial intelligence model for the technology behind the online retail giant's "Just Walk Out" checkout-free service for brick-and-mortar retailers. The new model further increases the accuracy of the previous machine learning capabilities of the previous technology and applies them to physical stores by applying them to cameras, sensors in shelves, and other information. For retailers the system works a lot faster and easier to install; for shoppers, it means that they can just supply their payment information, credit card or mobile wallet, on entry and shop. Upon exit, the payment for items taken is produced along with a receipt. Just Walk Out technology was piloted by Amazon in 2018 and used generative AI and machine learning technology at the time to examine shopper behavior at the time to examine shopper behavior with a series of models, Jon Jenkins, vice president of Just Walk Out technology, AWS applications explained during a press briefing. The technology has been used in Amazon Fresh and smaller Go-branded stores for years. Now Just Walk Out is also provided as a solution for third-party retailers to install in their shopping venues for cashier-less shopping. According to Jenkins, the technology is available in over 170 locations globally including airports, stadiums, universities and hospitals in the U.S., U.K. and Canada. Jenkins compared the new Just Walk Out model to large language models such as OpenAI's GPT-4o, which is multi-modal, which means it can take text, audio and video and output text, audio and video. "Instead of generating text or images, we generate receipts," Jenkins said. "And instead of the input information being snippets of text or questions, our input is video from cameras, weight signals from sensors on shelves and pictures of what comes out in someone's hand. It can also work with RFID chips embedded in products." This replaced the previous system which was a series of models each of which did a sub-task of this process where one model had to wait for the previous signal to resolve. One model had to track a person moving around the store, another had to wait to see someone reach into a shelf and yet another had to wait to see what was withdrawn. This was slow, tedious and needlessly complicated. In the new model, all of this happens at once as it is capable of combining all of these signals from sensors, cameras and radio-frequency chips with a 3D map of the store into one action. This means it can readily handle strange issues that shoppers create in a chaotic retail environment. For example, a shopper might pick up and put down multiple varieties of soda at a time. It's not uncommon for shoppers to pick up an item, walk to another part of the store and then put something down on a shelf where it doesn't belong - much to the chagrin of store employees. A freezer could fog up, blocking the camera's view of the items within. All of these complex situations can be handled more accurately by the new model, which can learn quickly from the sensors and cameras by watching the hands and items using computer vision and self-teaching algorithms. Jenkins was keen to point out that Just Walk Out technology also changes the nature of shopping as it could reduce shrink or shoplifting. "When you put Just Walk Out technology in a store what would have been theft - me taking a Snickers bar and jamming it in my pocket - that becomes a purchase." He also emphasized that although the technology makes heavy use of cameras to track the movement of customers around the store, it does not use or collect any biometric information to complete its tasks. "We don't use any biometrics at all. We don't even track what clothing you're wearing or anything like that," Jenkins said. For example, it's not uncommon for Just Walk Out to be deployed in stadiums where everyone is wearing the same jersey or ballcaps, which makes it difficult to identify people based on their outfits. "So, if you were using the visual appearance of what people looked like to determine who they are or where they were in the store, that would be a bad way to go about it." Instead, the system tracks people by their movements and the movement of their hands, he explained. The most important thing to the system is that that pair of hands "signed in" with a payment method at entry and that is mapped to them upon entry. In fact, the same person could leave the store - which would trigger a receipt and a payout - then re-enter the store with a different payment method causing the system to believe they were a whole new person.