AI Model Simulates Piano Playing to Reduce Injury Risk and Promote Inclusivity

AI could help reduce injury risk in pianists

Professional piano players spend countless hours at the keys, perfecting their craft. For people with smaller hands, this dedication can take a physical toll. Repeated stretching to reach distant keys in a chord can strain muscles and joints and may lead to tendonitis, carpal tunnel syndrome, and other injuries. Researchers at Stanford are working to understand the forces involved in the hand movements of elite-level piano players, and how those change with different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers presented an AI-trained model that can recreate the hand movements required to play complicated pieces of music and the physical stresses those hands are under. This is the first step in an effort to reduce the risk of long-term injuries in piano players and test solutions - such as narrower keyboards - that could make piano playing more equitable for everyone. "We would never expect a world-class athlete to compete with equipment that does not fit their body. Yet we ask pianists, particularly women, to adapt to a one-size-fits-all design that was never built with them in mind," said Elizabeth Schumann, the Billie Bennet Achilles Director of Keyboard Studies at Stanford and co-author on the paper. "We can use research like this to reshape the future of performance and make it more sustainable." The modern piano keyboard was designed in the 19th century with the average European man in mind. Today, an estimated 87% of adult women and 24% of adult men have hands that are smaller than ideal for the standard piano. Using traditional methods to study the impacts of smaller hands would require following a cohort of piano players for decades, and it could still be difficult to quantify injury risks or test solutions. Instead, the researchers have recorded the hand movements of professional piano players and are using artificial intelligence to predict how different sizes of hands would move to play new music. Schumann and her colleagues recruited 15 elite-level pianists to play a total of 10 hours of music while cameras filmed their hands from every angle. The researchers couldn't put any sensors on the players without potentially interfering with their performance, so they used advanced computer vision techniques to combine the videos and reconstruct the players' hand motions in three dimensions. They used additional processing to make sure that those movements were perfectly synced with the audio. "The quality of data that we were able to achieve is unprecedented," said Karen Liu, a professor of computer science at Stanford and lead author on the paper. "It's not just that the reconstructed 3D motions are accurate and clean; the quality of the performance itself is truly remarkable. We were able to capture diverse movements from a group of talented musicians performing at a high level - I have not seen a dataset like that in the past." Roucheng Wang, a graduate student in Liu's lab, and Pei Xu, a postdoctoral researcher, used this dataset to train a model to accurately generate new piano-playing data. When they gave the model sheet music it had never seen, such as Beethoven's "Für Elise," the model could provide the correct 3D hand movements needed to play the piece, using a simulated piano to generate sound. "I was just so stunned at how accurately this model could simulate elite-level technique," Schumann said. "It's really incredible." This model is just the first step for Liu, Schumann, and their colleagues. At the moment, the model can physically simulate hand movements, but it doesn't simulate the muscles and tendons that create those movements or the strain that they are subjected to. "We are not at the biomechanics level yet, where the actuation of the hands is generated by muscles - currently our model is more like a robot that directly generates torque at joints," Liu said. "Eventually, our model needs to be able to predict the tension level of muscles and the potential of injury." The researchers are currently working to add these layers to their model so that they can use it to help piano players - especially those with small hands - avoid injuries. They are also adapting their work to model the movements of other dexterous musicians. They have applied some of the same techniques to an existing dataset of hand motions of highly skilled guitar players. Their results, which were also presented recently at SIGGRAPH Asia 2024, showed that they could successfully model the distinct left- and right-hand movements needed to play a variety of both popular and classical music. "With high-quality data, we could model the 3D movements needed for other types of music performance as well," Liu said. "This isn't a model that is replacing people - we're working with musicians to help understand and solve problems. This project is about advancing people, and AI is just a tool for that."

AI could help reduce injury risk in pianists

Researchers at Stanford Engineering have developed an AI-trained model to accurately recreate the hand movements of elite-level pianists and the physical stresses they endure while playing. Professional piano players spend countless hours at the keys, perfecting their craft. For people with smaller hands, this dedication can take a physical toll. Repeated stretching to reach distant keys in a chord can strain muscles and joints and may lead to tendonitis, carpal tunnel syndrome, and other injuries. Researchers at Stanford are working to understand the forces involved in the hand movements of elite-level piano players, and how those change with different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers presented an AI-trained model that can recreate the hand movements required to play complicated pieces of music and the physical stresses those hands are under. This is the first step in an effort to reduce the risk of long-term injuries in piano players and test solutions -- such as narrower keyboards -- that could make piano playing more equitable for everyone. "We would never expect a world-class athlete to compete with equipment that does not fit their body. Yet we ask pianists, particularly women, to adapt to a one-size-fits-all design that was never built with them in mind," said Elizabeth Schumann, the Billie Bennet Achilles Director of Keyboard Studies at Stanford and co-author on the paper. "We can use research like this to reshape the future of performance and make it more sustainable." Capturing elite performance The modern piano keyboard was designed in the 19th century with the average European man in mind. Today, an estimated 87% of adult women and 24% of adult men have hands that are smaller than ideal for the standard piano. Using traditional methods to study the impacts of smaller hands would require following a cohort of piano players for decades, and it could still be difficult to quantify injury risks or test solutions. Instead, the researchers have recorded the hand movements of professional piano players and are using artificial intelligence to predict how different sizes of hands would move to play new music. Schumann and her colleagues recruited 15 elite-level pianists to play a total of 10 hours of music while cameras filmed their hands from every angle. The researchers couldn't put any sensors on the players without potentially interfering with their performance, so they used advanced computer vision techniques to combine the videos and reconstruct the players' hand motions in three dimensions. They used additional processing to make sure that those movements were perfectly synced with the audio. "The quality of data that we were able to achieve is unprecedented," said Karen Liu, a professor of computer science at Stanford and lead author on the paper. "It's not just that the reconstructed 3D motions are accurate and clean; the quality of the performance itself is truly remarkable. We were able to capture diverse movements from a group of talented musicians performing at a high level -- I have not seen a dataset like that in the past." Roucheng Wang, a graduate student in Liu's lab, and Pei Xu, a postdoctoral researcher, used this dataset to train a model to accurately generate new piano-playing data. When they gave the model sheet music it had never seen, such as Beethoven's "Für Elise," the model could provide the correct 3D hand movements needed to play the piece, using a simulated piano to generate sound. "I was just so stunned at how accurately this model could simulate elite-level technique," Schumann said. "It's really incredible." From robot joints to real muscles This model is just the first step for Liu, Schumann, and their colleagues. At the moment, the model can physically simulate hand movements, but it doesn't simulate the muscles and tendons that create those movements or the strain that they are subjected to. "We are not at the biomechanics level yet, where the actuation of the hands is generated by muscles -- currently our model is more like a robot that directly generates torque at joints," Liu said. "Eventually, our model needs to be able to predict the tension level of muscles and the potential of injury." The researchers are currently working to add these layers to their model so that they can use it to help piano players -- especially those with small hands -- avoid injuries. They are also adapting their work to model the movements of other dexterous musicians. They have applied some of the same techniques to an existing dataset of hand motions of highly skilled guitar players. Their results, which were also presented recently at SIGGRAPH Asia 2024, showed that they could successfully model the distinct left- and right-hand movements needed to play a variety of both popular and classical music. "With high-quality data, we could model the 3D movements needed for other types of music performance as well," Liu said. "This isn't a model that is replacing people -- we're working with musicians to help understand and solve problems. This project is about advancing people, and AI is just a tool for that."