Improving Sign Language Translation through Multimodal Language Alignment

Abstract

Over 5% of the world’s population experiences disabling hearing loss and many of themrely on sign languages as their primary means of communication. Despite the widespread useof sign languages, most modern communication technologies are designed primarily for spokenlanguage, leaving deaf signers at a significant disadvantage. Recent advances in artificialintelligence offer new opportunities to address these communication barriers.The main objective of this work was to design a new system for sign language translation.At its core is a novel video feature extraction model that combines both spatial and temporalinformation. In addition, the entire system supports pretraining through language alignment.