A Multimodal User Interface Component for an Augmented Reality Mobile User Guidance System

Abstract

In general, user interfaces should be intuitive, self-explanatory and adaptive to various user skills. Especially in augmented reality systems with complex interaction possibilities simple concepts are necessary to guide the user. Therefore, we extend the ideas of the traditional WIMP metaphor (Windows, Icons, Menus, Pointer) with the capabilities of multimodal interfaces where multiple ”human” communication channels are used as input data for navigation, orientation and interaction. In this work we will present the adaption of the user interface of an existing mobile augmented reality system for cultural heritage to multimodal interaction. We present the concepts that led us to the decision for the use of speech and capturing of hand movements by means of an inertial tracker, as well as the implementation aspects of the initial prototype. User evaluation trials will prove our approach.