Sony Patents AI-Powered Controller to Predict Player Inputs and Reduce Latency

Sony controller patent aims to predict your button presses using AI and a camera

Forward-looking: Nvidia and AMD have been using AI to cut down on input lag for a while now. A new patent from Sony also uses artificial intelligence to reduce latency, though it does so in a very unusual way: the technology is combined with a camera to predict button presses ahead of time. The patent in question, titled WO2025010132 - TIMED INPUT/ACTION RELEASE, was spotted by Tech4Gamers. It describes using machine learning to predict a player's movements before they complete them, which could theoretically reduce input lag. The AI model and an external sensor such as a camera would work in tandem. The camera watches the players fingers to see which button you're about to press and the AI uses this data to learn players' movements. Essentially, it could register a button press before the player has pressed, which should reduce latency. The system could also detect incomplete controller movements and determine the player's intended action, completing it on their behalf. "There can be latency between the user's input action and the system's subsequent processing and execution of the command," Sony writes. "This in turn results in delayed execution of the command and unintended consequences in the game itself." A possible alternative to the slightly creepy prospect of having a camera constantly pointed at your fingers is to integrate the sensor into the controller buttons themselves, potentially allowing them to register when a digit is hovering above. The majority of patents don't become real products, so there's no guarantee this one will ever move beyond the planning stage. However, Sony won plenty of praise for the PS5's excellent DualSense controllers, so it might want something equally impressive in the PlayStation 6. Moreover, the introduction of PlayStation Spectral Super Resolution (PSSR) in the PS5 Pro does introduce a tiny amount of latency (around 2ms), which Sony might want to address in its future consoles. Nvidia has championed latency-reducing technology with its Reflex tech. The latest version, Reflex 2, which was unveiled at CES, also uses predictive AI to decrease latency - by 75%, according to Nvidia. While Reflex 2 continues to synchronize CPU and GPU tasks, it also introduces a feature called Frame Warp. This updates rendered frames based on the most recent mouse inputs just before they are displayed. As the GPU renders a frame, the CPU calculates the next frame's camera position using the latest input data. Frame Warp then modifies the current frame to reflect this updated position, ensuring that on-screen actions correspond more closely with player inputs. The process involves a predictive inpainting technique that uses color and depth data from prior frames to fill any gaps created during the frame warp.

Sony patents include idea to lower input lag by using AI to predict which buttons you'll press next

Two new Sony patents have been spotted that would allow players to aim their DualSense like a gun, and use AI to predict player inputs. The first of these, spotted by GameRant, was filed in June last year and made public earlier this month. The patent is for a trigger attachment for the DualSense controller to add realism to gunplay. Players would theoretically hold the controller sideways with one controller arm, using the space between the R1 and R2 shoulder buttons as an aiming sight. The patent also suggests this could be paired with other hardware or accessories, including a VR headset. Though with PS VR2's Sense controllers I don't know why this would be necessary. What's more, the controller arms don't seem long enough to me to house player hands and a trigger system, while the weight of the controller would be uneven to hold up for long periods. The second patent is, perhaps, more interesting as it could be used to reduce lag in online games by using AI to predict player inputs ahead of time (thanks tech4gamers). The patent involves using a camera to view the player and controller, record actions, and let AI predict what inputs will be pressed next. Another option would be to use incomplete controller actions instead, allowing AI to guess players' intentions. Essentially this would help to reduce latency - the delay between player input and in-game action - which can be a major issue during online games. Other recent patents spotted by tech4gamers include using AI accounts to impersonate players and catch cheaters, and providing real-time in-game location information to assist frustrated players - as well as detecting erratic button mashing and verbal outbursts. Of course, these patents don't mean these ideas will necessarily come to fruition, but it's always interesting to see what companies may be working on.