SUNY Potsdam Professor Tracy Lipke-Perry Uses Video Motion Capture to Optimize Piano Performance
A classic use of video is to capture a performance that occurs very quickly in real time, and to slow it down to review the details of what is actually happening. Recently, motion capture has added a new dimension to this process, allowing for extremely detailed recording and analysis of movements. Professor Lipke-Perry and her colleagues conducted studies in 2017 and 2018 using this technology to analyze pianists’ movements. In 2017, she positioned eight digital motion capture cameras across a custom frame built to accommodate the dimensions of a standard keyboard instrument. She then placed 22 reflective markers on the hand of each pianist including the fingertip, each finger joint, and several on the wrist as position references. In this way, a three-dimensional tracking of each pianist’s movements was captured for analysis, giving each participant precise analysis of their body position and movements facilitating suggestions for improvements.
Pianists were shown their “skeleton” video track immediately after performing. The results, which were universal among pianists of a variety of levels of expertise, indicated that the timing of piano keystrokes varies whenever the thumb is involved. Identifying this commonality is important in working with pianists who might be recovering from injury. In that way, motion capture has the potential to identify performance aberrations at a very early stage but also to track recovery in ways that are not visible to the naked eye in real time. The implications are in repetitive stress injuries and the balance of coordinated muscle use.
Two exercise science students were part of the team in pilot work, capturing data, and the early stages of analysis. Due to the emerging nature of the technology, in particular the application at the piano, several weeks of pilot studies were necessary to determine (1) optimal camera placement, (2) optimal marker placement, and (3) the basic operation of the camera system and data capture software. This experience helped prepare the exercise students for other real-world research opportunities in terms of problem-solving and use of cutting-edge technology and interdisciplinary collaboration.
The 2018 study, pianists played the same short piece four times in different conditions. The first time, they all played the piece however they wanted. In the next three trials, pianists were asked to focus on: (A) their fingertips and creating short, detached-sounding notes (B) creating the feeling of a dance, and (C) the sound of a metronome keeping a steady beat. The difference in these conditions was comparing performance while focusing internally (on one’s own body) versus performance while focusing externally (on an idea/thought or alternatively on something in one’s environment). After numerous years of teaching piano in the studio, Dr. Lipke-Perry has observed that it is often challenging for pianists to articulate details related to their focus. Many pianists would finish playing a piece and not have any idea what they were focusing on. Naïve to the purpose of the study at the outset, but later knowing the premise as well as the basis of the study related to sports literature, pianists left with a definite idea about how they would approach “performing” going forward. Overwhelmingly, both the participants and Dr. Lipke-Perry’s colleagues observed more musical, fluid, and sensitive performances in the condition relating to the external focus, the idea of the dance. The movements themselves were very different as observed in the motion capture videos. Even if sound was removed, the differences in continuity and smoothness of the movements, as well as movement intent, were obvious. In fact, while focusing on the fingertips, not only were the performances less artistic, but they were also less accurate in terms of playing the correct notes and rhythms. The lesson for the participating pianists was overwhelming in terms of future application. They could improve, and in some cases, revolutionize their performances by focusing on an idea or musical thought.
All participants in these studies were part of a cutting-edge, real-world experience with transformative results, whether they were a pianist, scientist, or both. Application of digital motion technology is emerging and becoming more widely utilized as the benefits of movement analysis and healthy performing are realized.