UMBC researchers are discovering building blocks of hand motions, aiming to improve physical therapy for humans and find better ways to program robots. They’re turning to a novel source material for these gestures: classical Indian dance.
Fossil records suggest that between four and six million years ago, the hominin ancestors of modern humans first stood up and walked on two legs—thus freeing their hands. Those hands went on to craft humanity’s story arc: cradling babies, carrying food, fashioning and wielding weapons, carving the woodblocks used to print the first paper books, running over the keys of a piano in a Rachmaninoff concerto, and even planting a flag on the surface of the moon.
“Hands are incredibly important to humans,” says Ramana Vinjamuri, an associate professor of computer science and electrical engineering whose work has focused on understanding how the brain controls complex hand movements.
Vinjamuri personally witnessed the debilitating impact of loss of hand movement when his mother suffered a stroke in 2014. “The very hand that taught me how to draw, how to write—I saw that hand irrevocably paralyzed. It was really hard for the family.”
The experience motivated Vinjamuri to work on technologies that could help people regain lost motor functions or serve as robotic replacements for injured body parts. As part of the research, the team began searching for and cataloging the building blocks of hand motions. Further inspiration struck when Vinjamuri attended a scientific conference on the brain, hosted by the Indian Institute of Technology Mandi in the serene foothills of the Himalayas. While brainstorming ideas for a session of the conference focused on ways that ancient Indian traditions might be applied to modern problems, Vinjamuri conceived a novel approach to deriving these building blocks—from the structured hand gestures of Indian classical dance.
Ramana Vinjamuri in the Vinjamuri lab, stands with a Unitree bipedal robot.
A Complex and Versatile Instrument
Mitra, a humanoid robot produced by Invento Robotics, a company founded by UMBC alumnus Balaji Viswanathan, M.S ’06, Ph.D. ’23, computer science. The researchers programmed Mitra to make letters of the American Sign Language alphabet by combining the mudras-derived alphabets of movement, in this case making the letter E.
Take a moment to consider your hands. Including the wrist, each hand has 27 joints. Some of those joints, such as the carpometacarpal joint at the base of the thumb, can move in multiple ways, such as rotating, bending, and moving toward or away from the center of the palm. The full hand encompasses billions of possible unique combinations of movements.
“To study something as complex as the hand is fascinating,” says Parthan Olikkal, a longtime member of Vinjamuri’s lab who is currently working toward his Ph.D in computer science and is deeply involved in recent research efforts.
To get a grip on the complexities, the team has turned to a concept called kinematic synergies. First extensively explored in the mid-20th century by Russian physiologist Nikolai Bernstein, synergies are essentially building blocks of movement in which the brain simultaneously coordinates multiple joint movements to simplify complex motions.
The concept can be used to deconstruct a dazzling diversity of movements into a limited number of fundamental units, similar to how the hundreds of thousands of different words in the English language can be broken down into only 26 letters.
Vinjamuri and his students have been on a quest to discover the “alphabets” of human hand movements we’ve collectively learned through hundreds of dropped sippy cups, hours of handwriting practice, and the like. The hope is that the knowledge could then be used as a “hack”—to more effectively train ourselves and our robotic assistants in the future.
“The way we interact with our surroundings—the way we grasp objects or move through space—feels so natural and effortless. We often forget we stumbled and fell as children while our brains and bodies were learning to coordinate.”
— Ramana Vinjamuri, associate professor of computer science and electrical engineering
Natural Versus Structured Movements
Ashwathi Menon, co-caption of the Adaa Indian fusion dance team, stopped by the lab for a photo shoot in October. Here she appears on the computer screen as the team demonstrates how to use a simple camera and software system to recognize hand movements.
A small statue representing the Hindu god Shiva in the form of Nataraja, the cosmic dancer. Ramana Vinjamuri keeps the statue in his office. (Photo courtesy of Vinjamuri)
As part of their latest research on alphabets of hand movements, Vinjamuri and his students analyzed a dataset of 30 natural hand grasps. The movements are used for picking up objects ranging in size from large water bottles to tiny beads. The researchers found six synergies, akin to an alphabet of six letters, that when combined could account for nearly 99 percent of the variations in movements represented in the full dataset.
The first two synergies alone—specifically a movement in which all five fingers wrap around an object and a movement in which the index finger and thumb pinch together—could account for more than 90 percent of the variations. Learning (or relearning) those two movements would be essential to training a hand to pick up objects, the researchers say.
However, the team also says that studying natural grasps has limitations.
“Natural grasp is unique to the motor learning history of an individual,” says Olikkal. “So the way I do something might be completely different from another person.” The grasps also represented limited functionality, containing only a small subset of ways that a person might use their hands.
“Hand gestures are part of the storytelling. They are very precise. They can be used to point, to represent an animal, to represent praying—those are just some examples.”
— Ashwathi Menon, a junior and co-captain of UMBC’s Adaa Indian fusion dance team
In search of richer alphabets of movement, the researchers turned to dance, specifically an Indian classical dance form called Bharatanatyam, which originated in the southern Indian state of Tamil Nadu. The term Bharatanatyam is often explained as a mnemonic blend of words combining the concepts of emotion, melody, rhythm, and dance. The holistic art form employs a variety of hand gestures, called mudras, to drive the storytelling at its heart.
“Bharatanatyam is an intricate, linear, and structured dance form with a lot of precision,” says Ashwathi Menon, a UMBC junior majoring in bioinformatics and computational biology who is co-captain of the university’s Adaa Indian fusion dance team and who has been performing classical Indian dances since she was four years old. “Hand gestures are part of the storytelling. They are very precise. They can be used to point, to represent an animal, to represent praying—those are just some examples.”
“We noticed dancers tend to age super gracefully: They remain flexible and agile because they have been training,” says Vinjamuri. “That was a huge inspiration for us when we started looking for richer alphabets of movement. With dance, we are looking not just at healthy movement but super healthy. And so the question became, could we find a ‘superhuman’ alphabet from the dance gestures?”
Mitra’s hands use three types of motors for movement. The strongest motor, located at the shoulder, handles shoulder flexion and extension, while a medium-torque motor at the elbow controls elbow movements, and five servo motors, one for each digit, are used to control the fingers.
Chris Dollo (left), a senior computer science major and undergraduate researcher in the Vinjamuri lab, and Parthan Olikkal (right) work at the computer. Olikkal brought the concept of capturing hand movements using simple camera set-ups into the lab, a key step toward making cost-effective technologies.
Dance-Derived Alphabets of Movement
Ashwathi Menon demonstrates mudras, which are copied by an Inspire robotic hand. From top to bottom the mudras are: Ardhachandra, meaning “half moon;” Chandrakala, meaning “crescent moon;” and Tripataka, meaning “three parts of the flag.” The mudras can demonstrate various elements of a story, including weapons, trees, flowers, or concepts such as balance, unity, and beauty.
Using the same techniques they had deployed to deconstruct the 30 natural hand grasps, the research team also analyzed 30 single-hand mudras. They found six synergies that could account for around 94 percent of the mudras’ variations.
Crucially, the team then tested how well the six natural grasp-derived synergies could combine to construct unrelated hand motions—in this case 15 letters of the American Sign Language alphabet—compared to the mudras-derived synergies. The mudras synergies significantly outperformed the natural hand grasp synergies on that task.
“When we started this type of research more than 15 years ago, we wondered: Can we find a golden alphabet that can be used to reconstruct anything?” says Vinjamuri. “Now I highly doubt that there is such a thing. But the mudras-derived alphabet is definitely better than the natural grasp alphabet because there is more dexterity and more flexibility.”
Ultimately, Vinjamuri envisions coming up with libraries of task-specific alphabets that can be deployed depending on the needs, be it completing everyday household chores such as cooking or folding laundry, or something more complicated and precise, such as playing an instrument.
Robotic Helping Hands
Apart from advancing understanding of the fundamental roots of movement, the team has made great strides developing cost-effective and pragmatic methods of testing and implementing their ideas. When Vinjamuri first started the work, his team relied on motion-capture systems that required specialized gloves and other equipment. Now, the team uses a simple camera and software system to recognize, record, and analyze movements.
“Parthan brought the concept of capturing hand movements using cameras into the lab and really developed it,” Vinjamuri said. It’s an important contribution to ultimately making cost-effective technologies that people could use in their homes, he says, such as a virtual system to coach people through physical therapy sessions.
The team is also successfully developing techniques to “teach” robotic hands the alphabets of movements and how to combine them to make new hand gestures. The approach marks a departure from standard techniques of teaching robots to mimic hand gestures, and toward a method rooted in our understanding of how the human body and brain work.
Ashwathi Menon, left, demonstrates the Katakamukha mudra while a robotic hand mimics her gesture. The mudra is often used to represent actions like plucking flowers, holding a necklace, and pulling a bowstring.
“It’s called biomimetic learning,” says Vinjamuri. “We want to watch how a human body moves, how a hand moves and works, and we want to derive those principles and apply them to machines.”
The researchers are testing the techniques on a stand-alone robotic hand and a humanoid robot, each of which operates in a different way and requires a unique approach to translating the mathematical representations of synergies into physical movements.
“Once I learned about synergies, I became so curious to see if we could use them to make a robotic hand respond and perform the same way as a human hand,” says Olikkal. “Adding my own work to the research efforts and seeing the results has been gratifying.”
These moments of satisfaction in finding solutions to knotty problems will continue to propel the team’s voyage of discovery. They may even take a moment to celebrate their successes—perhaps with a fitting high-five.
Could A Dancing Robot Improve Humans’ Mental Health?
By Catherine Meyers • Photography by Kiirstn Pagan ’11
Searching for an alphabet of hand movements from the gestures in classical Indian dance is not the only way that art is inspiring and guiding new research in Ramana Vinjamuri’s lab. In a related project, Vinjamuri has teamed up with Andrea Kleinsmith, an associate professor in information systems who specializes in ways that computers can assess humans’ emotions, and Ann Sofie Clemmensen, an associate professor of dance, to explore whether and how dancing robots might offer humans new tools to improve their mental health. The research piggybacks off established practices of human-to-human dance/movement therapy, which can be used to treat some mental health challenges, such as schizophrenia, anxiety, and depression.
Dancers Sarah McHale ’24 and Juju Ayoub ’25 perform during the AccelNet meeting. The dance was a demonstration of a collaborative research project by UMBC faculty Ramana Vinjamuri, Andrea Kleinsmith, and Ann Sofie Clemmensen exploring stress reducing technology.
The spark for the interdisciplinary venture was struck when the College of Engineering and Information Technology launched a program to encourage faculty to explore collaborations across disciplines to tackle big challenges.
Together, Vinjamuri, Kleinsmith and Clemmensen developed a proposal to investigate questions such as whether the coordination in a person’s arms and legs could be a proxy measure of mental well-being, how existing dance therapy movements affect brain activity, and how a humanoid robot dance partner compares in effectiveness to a flesh-and-blood one.
“As a healthcare opportunity, dancing with a robot may sound weird at first,” notes Clemmensen. But, she says, people who are socially isolated or struggle with the stressors of human interactions might benefit from robot partners. “As humans, we project emotions on objects, but the objects do not judge back.”
In June, the team creatively demonstrated their progress to brain researchers and artists from around the world when they choreographed a technology-infused dance performance for the Movement, Music, and Brain Health National Science Foundation AccelNet meeting on the UMBC campus.
Two dancers—one representing a robot and the other a human—took turns moving around each other. Sensors monitored physiological signs of stress on the human dancer. As the dance progressed, the human was at first fearful, then curious, and finally happy—an ending the researchers hope their own project might one day also accomplish.
UMBC students and professors who worked on the project gather on the stage after the dance performance. From left to right are Viraj Janeja, Oritsejolomisan Mebaghanje, Golnaz Moharrer, Sruthi Sundharram, Parthan Olikkal, Ramana Vinjamuri, Juju Ayoub, Andrea Kleinsmith, Sarah McHale, and Ann Sofie Clemmensen.
Photo Credit: Niloufar Sarmast
