Saving sign language for the future by AI and machine learning
More than three hundred sign languages have been registered in the world. But despite the advances in Artificial Intelligence and language technology, only a small number of them have been supported by AI tools and language technology. In order to improve this, linguistics expert Mirella De Sisto, artificial intelligence specialist Dimitar Shterionov, and PhD student Lisa Lepp have been working on automatic recognition and translation of sign language through machine learning and computational linguistics approaches in the SignON project.
SignOn was a 3-year EU funded project led by Dr. Shterionov which ended in December 2023. The high-level objective of SignON is to create a mobile translation application. But, aside from the technical and scientific innovations in the field of AI and sign language translation, SignON was a unique project because of the close cooperation with the deaf community being one of the first large-scale projects to employ a co-creative process.
'There is no universal sign language'
Lisa Lepp explains how sign language originated: "Where deaf people meet, sign languages arise. There is no universal sign language: people from all over the world use different words, expressions and signs. Therefore, there are as many differences in sign languages as there are in spoken languages. Only 75 years ago scientific interest in sign language arose. It was the moment when researchers acknowledged the existence of sign language(s) with all their complexity and nuances. Influenced by globalization, which is creating a greater need for easy (cross-lingual) communication and better understanding of each other worldwide, including deaf and hearing people, the research efforts are now gaining momentum."
Lepp is herself deaf. She shares that she became deaf later in life: "I had my education in the hearing system. It was a long journey (laughs): through oral practical education, higher education, a Bachelor's degree in Dutch Language and Culture, a Master’s degree in General Linguistic with a specialization in Sign Languages, I eventually ended up participating in this study."
Only 75 years ago scientific interest in sign language arose. It was the moment when researchers acknowledged the existence of sign language(s) with all their complexity and nuances.
- Lisa Lepp (l).
Over the past three years, Shterionov and De Sisto have worked on the SignON project: Sign Language Translation Mobile Application and Communication. Its goal was to develop a mobile application for automatic translation between sign and spoken (written and spoken) languages. It focused on Irish, British, Dutch, Spanish and Flemish sign languages and on Irish (Gaelic), English, Dutch (including its Flemish variant) and Spanish spoken languages. Sign language has characteristics that distinguish it significantly from spoken language. Shterionov: "Sign language is conveyed through body, hand and lip movements as well as through facial expressions that can occur simultaneously.” De Sisto contrasts it to spoken language: “Spoken language is produced by sounds from the vocal tract and captured by ears in a series of sequential sounds.”
Technology development
Language technology for spoken languages has evolved with the properties and characteristics of spoken languages in mind. Automatic sign language recognition (SLR) is used in some applications nowadays, where users sign in front of a camera or upload a video and the system recognizes the different signs. Furthermore, machine translation (MT) techniques have been employed to translate between sign and spoken languages. However SLR or MT tools are far from optimal.
Along with simultaneity, another one of these challenges is related to the lack of officially accepted and widely adopted written form for sign languages. How do you identify and recognize signs to codify them as a language when you don’t have a writing standard to facilitate the recognition? The team notes that it is easier (for a computer) to record and process speech (and spoken language). Additionally, as with many other tasks solved with AI, tremendous amounts of data are necessary. Sign language recognition, translation and synthesis don’t have access to high volumes of (high) quality data. Along with the issues of data availability another concern is privacy. Privacy is a major issue when collecting data not only because people may not want to be recognizable on camera but also because personal features are involved in the model development.
Mobile application
In the 3 years of the SignON project, the team worked on addressing these features. The consortium of 17 partner organisations developed amongst others a signing avatar, called EVA, all packaged together in the SignON mobile application. It is an app that can assist in recognizing and translating (similar to Google Translate) between sign and spoken languages). But Lepp says: "It cannot make a perfect translation yet, but it can still help us.” Furthermore, a second application allows people to collect more data. “With the [machine learning] app, you can film yourself and gather data to feed the machine. This will help us progress in developing better models."
Collaboration is necessary to establish the right methodology and technological adjustments. We have achieved a lot in close collaboration with the deaf and hard of hearing community. In the past, people talked over their heads. This project is unique; it is a co-creation.
- Mirella De Sisto
But raw video data is not all. During the SignON-project, two side projects were initiated, led by De Sisto, with the aim of helping to reduce the scarcity of data suited for machine translation and machine learning in general. The results were two multimodal multilingual parallel corpora: one with Flemish Sign Language and written Dutch, and the other with Dutch, Flemish and Spanish Sign Language and written English, Dutch, Spanish and Irish (Gaelic). Linguist De Sisto adds that more knowledge about the sign languages involved is needed. "Collaboration is necessary to establish the right methodology and technological adjustments. We have achieved a lot in close collaboration with the deaf and hard of hearing community. In the past, people talked over their heads. This project is unique; it is a co-creation. I hope our research has laid the groundwork for further collaboration.” Indeed, the three researchers acknowledge the positive effects of co-creation which is one of the main lessons learned within SignON.
Future
The SignON project was concluded in December, but the researchers will continue. Lepp, with her PhD research, aims to advance the work in co-creation with the Dutch sign language community. "Indeed, in the past, research was executed from above, not from within the deaf community. Sign language was forbidden in Europe, by a law pronounced in Milan, in 1880. So, it was forbidden to use sign language in educational settings until 1980, and the oral method had the main focus. Research was only done by trained individuals and was not inclusive. Now, we are in the research group and working on an equal footing. I want to provide what the community requires.” Lepp’s PhD research aims to establish a collaborative, open and inclusive framework that will advance the development of SL tools and technology together with the main stakeholders, the deaf community. This will not only allow for expanding the repertoire of translation and interpreting tools but will also advance the documenting sign language, “Ultimately, I hope we can document sign language and make it accessible to everyone." Lepp adds. Lepp’s PhD project, with designated name SignUs, is in its first year but has already made the first steps towards establishing a co-creation network between the researchers and the deaf community.
SignUs is funded under the starters grant initiative; PhD student Ms Lisa Lepp, promoter Dr. Grzegorz Chrupala and daily supervisors Dr. De Sisto and Dr. Shterionov.