IMMERSIVE TV
Calculation of a depth map in a video
Integration of 3D Objects
Integration of Synthetic 3D Objects
Pilot for Scenario Living Room
Panoramic Visit in Scenario CAVE
Project co-funded by the European Fund for Regional Developement (FEDER) and the AVANZA2 program from Ministerio de Industria, Energía y Turismo (2008-2011) With Reference Number TSI-020302-2010-61
The purpose of this project is to research on the immersive digital content production and all the involved value chain (production, transmission and reception), developing the interactivity and immersivity further than current 3D, merely ‘bi-dimensional’. Thus, it is expected that the final user will experiment the feeling of belonging to the environment and being able as well to interact with it from any spatial direction.
Innovation in media consumption has been a constant evolution in the last decades: from color television to interactive television, from media digitization to the recent commercial development of High Definition, and the development of three-dimensional contents. One of the key aspects in the convergence experienced in the last years has been the evolution of virtual reality in an immersive way (generally focused on videogames) to conventional contents.
Results:
As a result of the project and for validation purposes, the design and implementation of two pilot scenarios has been carried out: CAVETM and LIVING ROOM.
The CAVETM scenario consists of an immersive experience based on a panoramic view of the city of Barcelona, using the I-Space system at CeDInt for its visualization. In this experience, the user can interact in real time with the scenario through 3D synthetic objects. This way, it is possible to realistically position and integrate them in the video scenes by taking into account the depth information and possible occlusions between objects. As part of the experience, other multimedia information has been included, such as 3D images and audio associated with the interactive content.
Figure 1. Panoramic view of Barcelona in the CAVETM scenario
Figure 2. Incorporating synthetic 3D objects (e.g. a building) in the video frame. The blue controller provides a visual feedback to the user to assist him/her during the insertion phase.
Regarding the LIVING ROOM scenario, a pilot test has been designed covering the whole chain value for the generation of immersive television content. In this case, the content consists of a football match recorded in high definition and stereoscopic format, coded and live broadcasted using a DVB modulator. The video stream can be received both at the main visualization terminal (HD TV) and any additional devices (e.g. tablet and smartphone) connected through the local area network. The video signal includes other additional content that contributes to enhance the user immersive experience, such as match statistics, commands acting on domotic devices and interactive 3D models synchronized with specific events of the match.
Figure 3. Pilot test of the LIVING ROOM exhibited during the JVRC 2012
The work of CeDInt Virtual Reality group has been mainly focused on the inclusion and integration of 3D synthetic content in the high definition video. In particular, different technologies and algorithms have been investigated and implemented, such as:
- Depth map estimation of stereoscopic video sources;
- Generation of panoramic video sequences from multiple captures corresponding to overlapping views; and,
- Real-time integration of 3D interactive elements by taking into account depth information for occlusions.
Figure 4. (Left): Original video frame of a football match; (Right): The computed depth-map
Figure 5. Integrating a 3D object (e.g. a car) in the video stream
Video 1. Demonstration of results in the CAVETM environment
Video 2. Demonstration of the results in the living room environment.
