Smart glove that translates sign languages into English, Chinese text wins contest

NUS Master students Gong He (left) and Huang Yunqi won first prize at the Huawei Tech4City Finals, on Sept 27, 2022. ST PHOTO: GIN TAY

SINGAPORE - A smart glove that instantly translates multiple sign languages into English or Chinese text is being developed in Singapore, potentially enabling deaf people to communicate without needing an interpreter.

The glove tracks the hand movement of sign language users with sensors that run along the four fingers and thumb, before interpreting the words in text via an app paired via Bluetooth.

The app also animates spoken English words that it records into sign language to help deaf people understand those who speak to them.

Titled See Your Voice, the project was the winning pitch at 2022's Huawei Tech4City Finals on Tuesday, beating 140 other teams of students in the inaugural competition that challenges students to use technology to tackle local issues.

Mr Gong He, 22, who is the business lead in the team of four, said they wanted to use technology to assist deaf people to communicate with others after the team learned about the daily struggles faced by some deaf friends.

More than 70 million people globally are deaf, while the number of individuals with hearing loss in Singapore is estimated to be around 500,000, Mr Gong cited in his presentation on Tuesday at Singapore Management University.

According to the Singapore Association for the Deaf, there are more than 5,400 people with hearing loss registered here.

According to its website, this is on the rise due to an ageing population.

Several deaf people interviewed said it was often challenging to communicate with those who could not speak in sign language, and that they preferred to keep to themselves or only with other signers, said Mr Gong.

There is also a shortage of sign language interpreters and a lack of translation services in mainstream media today, he added.

Mr Gong, who just completed a Master in Business Analytics at the National University of Singapore, said: "We may easily neglect our deaf friends in our community... We realised that there was a need by the deaf community to be understood and thought we could contribute with technology."

The passion project started in 2020 among Mr Gong, NUS data science and machine learning student Sun Shizhuo, 23, NUS mechanical engineering student Huang Yunqi, 24, and computer science PhD student Xu Xuanqi, 22, from Peking University. 

Using self-developed algorithms, they trained the glove to detect the hand movements of 8,000 words in various sign languages, such as the Chinese, American and Singapore versions, as instructed by the user. 

The word count and number of languages recognised is expected to grow as they work with Huawei to bolster the system’s machine learning capabilities.

Mr Huang said the glove is expected to be rolled out to some deaf communities here by the end of the year.

Contest judge Christina Lee, founder and chief executive of sustainability consultancy Global Green Connect, said the judges, which includes former foreign minister George Yeo, were sold by the project's social impact.

Ms Lee told The Straits Times: "The message was clear...people will see that the returns on investments here is not just money, but the social impact of the gloves."

For winning, the team was awarded $10,000 and will be fast-tracked into Huawei's Seeds of the Future programme, which sends young people from around the world for tech-related training in China.

In second place, the judges awarded a pitch to build a mobile app that can guide commuters with early-stage dementia around public areas like train stations.

In third place was a consumer app to support heritage businesses like local craftsmen and cobblers in curated listings for users.

Join ST's Telegram channel and get the latest breaking news delivered to you.