Study explores foot-based controls for augmented reality systems

Imagine controlling apps with your feet while you walk. This concept is the focus of new research which explores using gait gestures—intentional variations in how you walk—as controls for augmented reality (AR) devices.

The study, “Gait gestures: examining stride and foot strike variation as an input method while walking”, authored by Tsai, Vogel and Waterloo researchers Ryan Yen and Daekun Kim was published in the proceedings of UIST 2024.

“There’s a long history of using feet to control machines. For example, the pedals on the car, but very little research has been done into using the way we walk as an input for a device,” said Ching-Yi Tsai, the lead author on the study and a former visiting scholar at the University of Waterloo David R. Cheriton School of Computer Science.

The idea emerged during the pandemic when Waterloo professor of computer science Daniel Vogel, frustrated by having to stop and use his phone with cold fingers while walking to get coffee, wondered if there could be a way to place orders without pausing. This led to a study where volunteers tested 22 different foot motions, rating them on ease of movement, compatibility with walking, and social acceptability.

“Extreme movements like dance steps or a jump would likely be easy for a system to recognize, but these might be harder to perform, and they would deviate too far from normal walking for people to feel comfortable doing them in public,” Vogel said.

The research identified seven optimal gait gestures. In a follow-up study, participants used an AR headset displaying a simple menu overlaid with the real world.

They tested these gestures to operate a music player, order coffee, and answer calls. The team remotely triggered commands, as the corresponding AR technology is still in development. A proof-of-concept recognizer was also created, achieving 92% accuracy in identifying the gestures.

“We aren’t at a point yet where AR headsets are widely used,” Tsai said.