Autonomous robots require sophisticated perception systems to manipulate objects. In this context, cameras denote a rich source of information and are also precise and cheap. Our novel sensor approach uses cameras together with robust image processing algorithms to measure forces and other haptic data. 

The sensor system bases on flexible rubber foam without any internal electronics, which is attached on the fingers of a gripper. This flexible material shows a characteristic deformation during contact. The camera observes the foam and image processing algorithms detect the foam’s deformation from the camera images. Finally, our sensor software calculates haptic contact data during the grasp procedure.

This video shows the capabilities of this so-called visuo-haptic sensor in a sorting task. A linear robot with a two-finger gripper sorts plastic bottles based on their compliance. During the grasp procedure, the visuo-haptic sensor measures the contact forces, the pressure distribution along the gripper finger, finger position, the object deformation, and also estimates the object compliance as well as object properties like, e.g., shape and size. The novelty lies in the fact that all this data is extracted from camera images, i.e., the robot is “feeling by seeing”.

More information: https://www.ei.tum.de/lmt/forschung/ausgewaehlte-projekte/bmwi-exist-rovi/

This demonstration presents a novel approach to display textures via electrovibration. We collect acceleration data from real textures using a sensorized tool-tip during controlled scans and captured images of the textures. We display these acceleration data using a common electrovibration setup.

If a single sine wave is displayed, it can be observed that spectral shifts occur. This effect originates from the electrostatic force between the finger pad and the touchscreen. According to our previous observations, when multiple sine waves are displayed interferences occur and acceleration signals from real textures may not feel perceptually realistic.We propose to display only the dominant frequencies from the real texture signal, considering the JND of frequencies, to mitigate this observation. During the demo session, we will let the attendees feel the differences between previously recorded texture signals and our proposed frequency-reduced texture signals. Moreover, the effect of different amounts of dominant frequencies will be shown to the attendees.

This video presents a content-based surface material retrieval (CBSMR) system for tool-mediated freehand exploration of surface materials. We demonstrate a custom 3d printed sensorized tool and a common smartphone using an android-based surface classification app that are able to classify among a set of surface materials without relying on explicit scan force and scan velocity measurements.

The systems rely on features representing the main psychophysical dimensions of tactile surface material perception, namely friction, hardness, macroscopic roughness, microscopic roughness and warmth from our prior work. We use a haptic database consisting of 108 surface materials as reference for a live demonstration of both devices for the proposed task of surface material classification during the conference.

This demo demonstrates the prototype of a novel tactile input/output device for the exploration of object surface materials on a computer. Our proposed tactile computer mouse is equipped with a broad range of actuators to present various tactile cues to a user, but preserves the input capabilities of a common computer mouse.

We motivate the design and implementation of our device by the insights of related work, which reveals the five major tactile dimensions relevant for human haptic texture and surface material perception and provide tactile feedback for these tactile dimensions, namely microscopic/macroscopic roughness, friction, hardness and warmth. We use a haptic database from our prior work to provide the necessary signals (e.g., acceleration, image or sound data) for the recreation of surface materials and display them using our surface material rendering application.

This is a live demonstration of the visual indoor localization & navigation system developed for the project NAVVIS at the Institute for Media Technology at Technical University Munich.

The video shows our lab demonstrator running on an Android phone. By analyzing the camera images for distinctive visual features, the position and orientation of the smartphone is recognized and displayed on the map. For this demonstration, only visual information has been used - other localization sources have not been used to improve the vision-based results. In order to compare localization accuracy, Android's network-based position estimate (Wifi and cellular networks) is displayed.

The approach used in this demonstration of a vision-based localization system employs a novel combination of a content-based image retrieval engine and a method to generate virtual viewpoints for the reference database. In a preparation phase, virtual views are computed by transforming the viewpoint of images that were captured during the mapping run. The virtual views are represented by their respective bag-of-features vectors and image retrieval techniques are applied to determine the most likely pose of query images. As virtual image locations and orientations are decoupled from actual image locations, the system is able to work with sparse reference imagery and copes well with perspective distortion.

Publications

R. Huitl, G. Schroth, S. Hilsenbeck, F. Schweiger, E. Steinbach,
Virtual Reference View Generation for CBIR-based Visual Pose Estimation,
ACM Multimedia 2012, Nara, Japan, November 2012.

This video shows the TUM Indoor Viewer, a web-based panorama image and pointcloud viewer, developed for the NAVVIS project at the Institute for Media Technology at TUM. The viewer provides interactive browsing of more than 40,000 images from the TUMindoor dataset, recorded at the main site of Technische Universität München. The floor plans created by our mapping trolley are displayed on top of a regular maps layer (Google Maps, Satellite or Hybrid). Colored circles on the map and in the 3D view indicate locations where images are available for browsing.

The first part of the video shows views rendered from the panorama images recorded by the Ladybug3 camera system. The following part also shows the pointclouds that have been constructed from laser scanner data, overlaying them on top of the panorama images. Note that alignment between the image and the pointcloud is accurate only when the viewpoint is exactly at the location where the panorama images have been recorded.

Publications

R. Huitl, G. Schroth, S. Hilsenbeck, F. Schweiger, E. Steinbach,
TUMindoor: an extensive image and point cloud dataset for visual indoor localization and mapping,
IEEE International Conference on Image Processing (ICIP 2012), Orlando, FL, USA, September 2012.

This video demonstrates the capabilities of content based image retrieval (CBIR) algorithms when applied to mobile multimedia search. In this scenario we identify various CDs in real-time using low resolution video in a database of more than 390,000 CD covers. 

To achieve the low query times required for instant identification of the CD, transmission delay is minimized by performing the feature quantization on the mobile device and sending compressed Bag-of-Features (BoF) vectors. To cope with the limited processing capabilities of mobile devices, we introduced the Multiple Hypothesis Vocabulary Tree [1], which allows us to perform the feature quantization at very low complexity. A further increase of retrieval performance is accomplished by integrating the probability of correct quantization in the distance calculation. By achieving an at least 10 fold speed up with respect to the state-of-the-art, resulting in 12 ms for 1000 descriptors on a Nexus One with a 1 GHz CPU, mobile vision based real-time localization becomes feasible. In combination with the feature extraction as proposed by Takacs et. al [2], which takes 27 ms per frame on a mobile device, extraction and quantization of 500 features can be performed at 30 fps. 

Publications

G. Schroth, A. Al-Nuaimi, R. Huitl, F. Schweiger, E. Steinbach,
Rapid Image Retrieval for Mobile Location Recognition,
IEEE International Conference on Acoustics, Speech and Signal Processing, Prague, Czech Republik, Mai 2011.

Recognizing the location and orientation of a mobile device from captured images is a promising application of image retrieval algorithms. Matching the query images to an existing georeferenced database like Google Street View enables mobile search for location. Due to the rapidly changing field of view of the mobile device caused by constantly changing user attention, very low retrieval times are essential. These can be significantly reduced by performing the feature quantization on the handheld and transferring compressed Bag-of-Feature vectors to the server. To cope with the limited processing capabilities of handhelds, the quantization of high dimensional feature descriptors has to be performed at very low complexity. To this end, we introduced the Multiple Hypothesis Vocabulary Tree (MHVT) as a step towards real-time mobile location recognition. Our experiments demonstrate that our approach achieves query times reduced by up to a factor of 10 when compared to the state-of-the-art. 

This algorithm is part of the visual location recognition framework developed at the Institute for Media Technology, which is demonstrated in the video.

Publications

G. Schroth, A. Al-Nuaimi, R. Huitl, F. Schweiger, E. Steinbach,
Rapid Image Retrieval for Mobile Location Recognition,
IEEE International Conference on Acoustics, Speech and Signal Processing, Prague, Czech Republik, Mai 2011.

We work on a flexible in-vehicle Human-Machine-Interface (HMI) that is based on a service-oriented architecture (SOA) enriched by graphical information. Our approach takes advantage of standard web technologies and targets at a solution that can be flexibly deployed in various application scenarios as, e.g., for in-vehicle driver-assistance systems, in-flight entertainment systems or generally spoken, systems which allow multiple persons to interact with an HMI. 

Furthermore, we consider the integration of consumer electronic devices. The proposed system benefits from its SOA-driven, generic approach and comes along with advantages like platform-independency, scalability, faster time to market and lower price compared to application-specific approaches. 
In this video we show an excerpt of some of our new improvements in the area of HMI Generation. 

This work is supported, in part, by the BMBF funded research project SEIS

This video is a compilation of the games developed during the course "Projektpraktikum Multimedia" in the winter term 2009/10. 

Besides the haptic input and force feedback, an important aspect of the developed games is the need for cooperation among the participating players in order to achieve a common goal. 
The haptic controller used in this video is the Phantom Omni by Sensable Technologies.

With recent advances in telerobotics and the wide spread of Internet communication, a flexible framework for initiating, handling and terminating Internet-based telerobotic sessions becomes necessary. In our work, we explore the use of standard Internet session and transport protocols in the context of telerobotic applications. Session Initiation Protocol (SIP) is widely used to handle multimedia teleconference sessions with audio, video or text, and provides many services advantageous for establishing flexible dynamic connections between heterogeneous haptic interfaces and telerobotic systems. We apply this paradigm to the creation and negotiation of haptic telepresence sessions and propose to extend this framework to work with the haptic modality. We introduce the notion of a "haptic codec" for transforming haptic data into a common, exchangeable data format, and easy integration of haptic telepresence data reduction and control approaches. We further explore using the Real-Time Transport Protocol (RTP), a standard protocol for audio/video streaming media, for transport of teleoperation data. To demonstrate the proposed session framework a proof-of-concept system was implemented. Software for the prototype is based on the Open Phone Abstraction Library (OPAL). OPAL's media handling routines were extended to include a haptics media type, alongside audio and video media types. A haptic device interface wrapper was developed using the OpenHaptics API, and the "haptic raw" media format was linked to the device with OPAL's device interface API. The demonstrator successfully established haptics, audio, and video media streams, transmitting haptics data at close to 1kHz.