Perception-Based Integration of Vision and Touch in Virtual Environments

Abstract

The technological progresses that were achieved concerning Virtual Reality displays for the eyes of the user (head mounted displays, large projection screens, workbenches, CAVE, etc) as well as for his/her hands (forcefeedback arms or joysticks, exoskeleton gloves, tactile matrices, etc) allow today to simulate visual and haptic sensations that are very close from sensations generated by the visualization or interaction with real objects, in real situations. However, even though the realism of each one of these stimuli taken separately improves regularly and spectacularly, the coupling and the combination of multiple stimulations on different sensory channels raise numerous novel and complex questions for virtual reality applications. For instance, how to share and spread the sensory information of the virtual world on the multiple sensory channels at disposal? How to synchronize the separated sources of sensory information spatially and temporally? Are there any preferential combinations of sensory information? More generally, how do humans perceive and integrate information provided by different sensory channels? What are the perceptual and cognitive mechanisms that are involved? Can we take advantage from them when designing multi-sensory virtual environments? In addition to these numerous questions, a classical perceptual phenomenon can also strike our attention: the sensory illusions. Sensory illusions illustrate the plasticity of our brain and reveal its surprising and regular mistakes of interpretation of sensory information. The question of sensory illusions is a central question in virtual realitywhich can be seen as a vast operation aiming at generating the illusion of another reality, of another tangible universe. During the past decade, we have developed and proposed a novel concept of haptic feedback in virtual reality called 'pseudo-haptic feedback'. Pseudo-haptic feedback is meant to simulate haptic sensations in virtual environments using properties of the human perception. Pseudo-haptic feedback uses visual feedback and borders on sensory illusion. It has been already used to simulate various haptic properties such as: stiffness or mass of virtual objects, texture of images, etc. It was also successfully implemented in various applications, such as within a virtual environment for the vocational training to milling machines, or a medical simulator for the training to regional anaesthesia procedures. In this talk, we will first depict related work in the field of human perception, integration of vision and touch, and haptic illusions. Then we will report on experiments and on haptic properties that have been simulated to date using pseudo-haptic feedback. We will assess the potential applications of pseudo-haptic feedback in virtual reality and we will draw lessons from our past experience in implementing and evaluating visuo-haptic virtual environments.