Ann Arbor Times

A team of computer scientists at the University of Michigan has developed a new digital hand software, HandProxy, that allows users to control virtual and augmented reality environments using voice commands. The technology aims to improve convenience and accessibility in VR and AR by enabling hands-free interaction with digital spaces.

HandProxy enables users to direct a virtual hand to perform tasks such as grabbing objects, moving windows, and making gestures through spoken instructions. The software leverages GPT-4o, the AI model behind ChatGPT, to interpret complex commands like “clear the table” without needing detailed step-by-step instructions. This flexibility marks an advancement over current VR voice-command systems, which are generally limited to simple or predefined tasks.

“Mobile devices have supported assistive technologies that enable alternative input modes and automated user-interface control, including AI-powered task assistants like Siri. But such capabilities are largely absent in VR and AR hand interactions,” said Anhong Guo, Morris Wellman Faculty Development Assistant Professor of Computer Science and Engineering at the University of Michigan.

“HandProxy is our attempt to enable users to fluidly transition between multiple modes of interaction in virtual and augmented reality, including controllers, hand gestures, and speech,” Guo added.

While VR technology has focused on physical realism through hand-tracking cameras and gloves, this approach can be impractical for users in limited spaces or those with disabilities. People with motor impairments or chronic illnesses often find traditional VR controls challenging or inaccessible. HandProxy offers an alternative by providing voice-controlled interaction, which could make VR more inclusive for these users.

“If there is any built-in physics, which is true for most games and VR apps, HandProxy can interact with it,” said Chen Liang, a doctoral student at the University of Michigan and first author of the study. “Our virtual hand gives the same digital signal as the user’s hand, so developers don’t have to deliberately add something into their programs just for our system.”

In a study involving 20 participants, users were able to replicate tasks shown in a demo video and explore HandProxy’s capabilities. Some appreciated being able to communicate naturally with the virtual hand, while others saw potential for using it in more abstract ways beyond physical tasks.

“It could act like an agent, where a user gives it a high-level command, like ‘organize my workspace,’ and it finds a way to sort and close all your open windows,” Liang said.

The study found that while HandProxy successfully completed most of the 781 tasks requested by participants within one to four attempts, it failed at 64 tasks due to misunderstandings or ambiguous instructions. The research team is working on improving the system’s ability to interpret unclear speech. One participant suggested allowing the virtual hand to ask clarifying questions.

The team has filed for patent protection and is seeking partners for commercialization. The project received funding from the University of Michigan.