Framework for automated customer service in sign language

Abstract

Deaf people need the help of an interpreter in formal relations, such as visiting offices or medical institutions. We present a new framework for building systems for sign language interaction, which can provide basic automated customer service for the deaf. The framework covers all steps required to build such a system from scratch - the acquisition of scenario-specific corpora, extraction of features, training of models, recognition, user interface, integration and configuration of the final application. The usability of the framework has been evaluated by creating a proof-of-concept system for automated scheduling of doctor's appointments in sign language. The results indicate that the process of building a sign language interaction system with our framework is relatively quick and simple. Recognition efficiency was evaluated as well and proved to be sufficient for practical use.