Looking for publications? All our latest publications are listed here, and you can also use our search functions to help find what you are looking for. Power users might also want to consider searching on the EPFL Infoscience site which provides advanced publication search capabilities.

Events-to-Video: Bringing Modern Computer Vision to Event Cameras

Authors: Rebecq, Henri; Ranftl, René; Koltun, Vladen; Scaramuzza, Davide

 

Event cameras are novel sensors that report brightnesschanges in the form of asynchronous “events” instead ofintensity frames. They have significant advantages overconventional cameras: high temporal resolution, high dy-namic range, and no motion blur. Since the output of eventcameras is fundamentally different from conventional cam-eras, it is commonly accepted that they require the devel-opment of specialized algorithms to accommodate the par-ticular nature of events. In this work, we take a differ-ent view and propose to apply existing, mature computervision techniques to videos reconstructed from event data.We propose a novel recurrent network to reconstruct videosfrom a stream of events, and train it on a large amountof simulated event data. Our experiments show that ourapproach surpasses state-of-the-art reconstruction meth-ods by a large margin (>20%) in terms of image qual-ity. We further apply off-the-shelf computer vision algo-rithms to videos reconstructed from event data on taskssuch as object classification and visual-inertial odometry,and show that this strategy consistently outperforms algo-rithms that were specifically designed for event data. Webelieve that our approach opens the door to bringing theoutstanding properties of event cameras to an entirely newrange of tasks. A video of the experiments is available at https://www.youtube.com/watch?v=IdYrC4cUO0I

Reference

  • Presented at: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, USA, 2019
  • Read paper
  • Date: 2019
Posted on: May 31, 2019

Harnessing the Rheological Properties of Liquid Metals To Shape Soft Electronic Conductors for Wearable Applications

Authors: Hirsch, Arthur; Dejace, Laurent; Michaud, Hadrien O.; Lacour, Stéphanie P.

 

Emerging applications of the Internet of Things in healthcare, wellness, and gaming require continuous monitoring of the body and its environment, fueling the need for wearable devices able to maintain intimate, reliable, and unobtrusive contact with the human body. This translates in the necessity to develop soft and deformable electronics that match the body’s mechanics and dynamics. In recent years, various strategies have been proposed to form stretchable circuits and more specifically elastic electrical conductors embedded in elastomeric substrate using either geometrical structuring of solid conductors or intrinsically stretchable materials. Gallium (Ga)-based liquid metals (LMs) are an emerging class of materials offering a particularly interesting set of properties for the design of intrinsically deformable conductors. They concomitantly offer the high electrical conductivity of metals with the ability of liquids to flow and reconfigure. The specific chemical and physical properties of Ga-based LMs differ fundamentally from those of solid conductors and need to be considered to successfully process and implement them into stretchable electronic devices. In this Account, we report on how the key physical and chemical properties of Ga-based LMs can be leveraged to enable repeatable manufacturing and precise patterning of stretchable LM conductors. A comprehensive understanding of the interplay between the LM, its receiving substrate chemistry and topography, and the environmental conditions is necessary to meet the reproducibility and reliability standards for large scale deployment in next-generation wearable systems. In oxidative environments, a solid oxide skin forms at the surface of the LM and provides enough stiffness to counterbalance surface tension, and prevent the LM from beading up to a spherical shape. We review techniques that advantageously harness the oxide skin to form metastable structures such as spraying, 3D printing, or channel injection. Next, we explore how controlling the environmental condition prevents the formation or removes the oxide skin, thereby allowing for selective wetting of Ga lyophilic surfaces. Representative examples include selective plating and physical vapor deposition. The wettability of LMs can be further tuned by engineering the surface chemistry and topology of the receiving substrate to form superlyophobic or superlyophilic surfaces. In particular, our group developed Ga-superlyophilic substrates by engineering the surface of silicone rubber with microstructures and a gold coating layer. Thermal evaporation of Ga on such engineered substrates allows for the formation of smooth LM films with micrometric thickness control and design freedom. The versatility of the available deposition techniques facilitates the implementation of LM conductors in a wide variety of wearable devices. We review various epidermal electronic systems using LM conductors as interconnects to carry power and information, transducers and sensors, antennas, and complex hybrid (soft-rigid) electronic circuits. In addition, we highlight the limitations and challenges inherent to the use of Ga LM conductors that include electromigration, corrosion, solidification, and biocompatibility.

Reference

Posted on: May 31, 2019

Focus Is All You Need: Loss Functions For Event-based Vision

Authors: Gallego, Guillermo; Gehrig; Mathias; Scaramuzza, Davide

 

Event cameras are novel vision sensors that output pixel-level brightness changes (“events”) instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches. We call them focus loss functions since they have strong connections with functions used in traditional shape-from-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset,and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks:rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras.

Reference

  • Presented at: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 2019
  • Read paper
  • Date: 2019
Posted on: May 31, 2019

Event-based Vision: A Survey

Authors: Gallego, Guillermo; Delbruck, Tobi; Orchard, Garrick; Bartolozzi, Chiara; Taba, Brian; Censi, Andrea; Leutenegger, Stefan; Davison, Andrew; Conradt, Joerg; Daniilidis, Kostas; Scaramuzza, Davide

 

Event cameras are bio-inspired sensors that work radically different from traditional cameras. Instead of capturing images at a fixed rate, they measure per-pixel brightness changes asynchronously. This results in a stream of events, which encode the time, location and sign of the brightness changes. Event cameras posses outstanding properties compared to traditional cameras: very high dynamic range (140 dB vs. 60 dB), high temporal resolution (in the order of microseconds), low power consumption, and do not suffer from motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as high speed and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

Reference

Posted on: May 31, 2019

Optimal integration of intraneural somatosensory feedback with visual information: a single-case study

Authors: Risso, G.; Valle, G. ; Iberite, F. ; Strauss, I.; Stieglitz, T.; Controzzi, M.; Clemente, F.; Granata, G.; Rossini, P. M.; Micera, S.; Baud-Bovy, G.

 

Providing somatosensory feedback to amputees is a long-standing objective in prosthesis research. Recently, implantable neural interfaces have yielded promising results in this direction. There is now considerable evidence that the nervous system integrates redundant signals optimally, weighting each signal according to its reliability. One question of interest is whether artificial sensory feedback is combined with other sensory information in a natural manner. In this single-case study, we show that an amputee with a bidirectional prosthesis integrated artificial somatosensory feedback and blurred visual information in a statistically optimal fashion when estimating the size of a hand-held object. The patient controlled the opening and closing of the prosthetic hand through surface electromyography, and received intraneural stimulation proportional to the object’s size in the ulnar nerve when closing the robotic hand on the object. The intraneural stimulation elicited a vibration sensation in the phantom hand that substituted the missing haptic feedback. This result indicates that sensory substitution based on intraneural feedback can be integrated with visual feedback and make way for a promising method to investigate multimodal integration processes.

Reference

  • Published in: Scientific Reports (9, 7916)
  • DOI: 10.1038/s41598-019-43815-1
  • Read paper
  • Date: 2019
Posted on: May 31, 2019

Pedicle screw navigation using surface digitization on the Microsoft HoloLens

Authors: Liebmann, Florentin; Roner, Simon; von Atzigen, Marco; Scaramuzza, Davide; Sutter, Reto; Snedeker, Jess; Farshad, Mazda; Fürnstahl, Philipp

 

Purpose: In spinal fusion surgery, imprecise placement of pedicle screws can result in poor surgical outcome or may seriously harm a patient. Patient-specific instruments and optical system have been proposed for improving precision through surgical navigation compared to free-hand insertion. However, existing solutions are expensive and cannot provide in situ visualizations. Recent technological advancement enabled the production of more powerful and precise optical see-through head-mounted displays for the mass market. The purpose of this laboratory study was to evaluate whether such a device is sufficiently precise for the navigation of lumbar pedicle screw placement.
Methods: A novel navigation method, tailored to run on the Microsoft HoloLens, was developed. It comprises capturing of the intra operatively reachable surface of vertebrae to achieve registration and tool tracking with real-time visualizations without the need of intraoperative imaging. For both, surface sampling and navigation, 3D printable parts, equipped with fiducial markers, were employed. Accuracy was evaluated within a self-built setup based on two phantoms of the lumbar spine. Computed Tomography (CT) scans of the phantoms were acquired to carry out preoperative planning of screw trajectories in 3D. A surgeon placed the guiding wire for the pedicle screw bilaterally on ten vertebrae guided by the navigation method. Postoperative CT scans were acquired to compare trajectory orientation (3D angle)and screw insertion points (3D distance) with respect to the planning.
Results: The mean errors between planned and executed screw insertion were 3.38±1.73◦ for the screw trajectory orientation and 2.77±1.46 mm for the insertion points. The meantime required for surface digitization was 125±27s.
Conclusions: First promising results under laboratory conditions indicate that precise lumbar pedicle screw insertion can be achieved by combining HoloLens with our proposed navigation method. As a next step, cadaver experiments need to be performed to confirm the precision on real patient anatomy.

Reference

  • Published in: International Journal of Computer Assisted Radiology and Surgery (Volume: 14, Issue: 7)
  • DOI: 10.1007/s11548-019-01973-7
  • Read paper
  • Date: 2019
Posted on: May 31, 2019

Bi-Modal Control of Vacuum-Powered Soft Pneumatic Actuators with Embedded Liquid Metal-Based Strain Sensitive Skin

Authors: Robertson, M. A.; Dejace, L. M.; Lacour, S.; Paik, J.

 

Soft robotic systems are composed of active and passively deformable structures which are intrinsically compliant, flexible, and elastic. Although these features offer benefits of adaptability, robustness, and safety, controlling these types of robots is a significant challenge, in part from the difficulty of obtaining feedback from sensors which provide state information without hindering the advantageous material properties which grant these systems their unique mechanical behavior. We demonstrate here the first integration of a flexible, stretchable, liquid metal-based strain sensor with vacuum powered soft pneumatic actuators (V-SPAs) for simultaneous controlled feedback of the soft actuators as well as user input and soft robotic device interaction. The soft sensors which are encapsulated within a Polydimethylsiloxane (PDMS) membrane are directly embedded in the outer body skin of the soft actuators, and can be used to correlate the deformation of the body under vacuum actuation to overall actuator strain or to detect external disturbances. This information is used to compute and control the angle of a rotational 3-DoF actuator module, as well as detect implicit user input control signals by direct interaction without the need for an external control interface. The dual use of embedded sensing shown in this work provides a fundamental strategy for soft collaborative robot applications.

Reference

  • Presented at: IEEE International Conference on Soft Robotics, Seoul, South Korea
  • Detailed record: Infoscience
  • Read paper
  • Date: April 2019
Posted on: May 10, 2019

Harnessing the Rheological Properties of Liquid Metals to Shape Soft Electronic Conductors for Wearable Applications

Authors: Hirsch, A. E.; Dejace, L. M.; Michaud, H. O.; Lacour, S.

 

Emerging applications of the Internet of Things in healthcare, wellness, and gaming require continuous monitoring of the body and its environment, fueling the need for wearable devices able to maintain intimate, reliable, and unobtrusive contact with the human body. This translates in the necessity to develop soft and deformable electronics that match the body’s mechanics and dynamics. In recent years, various strategies have been proposed to form stretchable circuits and more specifically elastic electrical conductors embedded in elastomeric substrate using either geometrical structuring of solid conductors or intrinsically stretchable materials. Gallium (Ga)-based liquid metals (LMs) are an emerging class of materials offering a particularly interesting set of properties for the design of intrinsically deformable conductors. They concomitantly offer the high electrical conductivity of metals with the ability of liquids to flow and reconfigure. The specific chemical and physical properties of Ga-based LMs differ fundamentally from those of solid conductors and need to be considered to successfully process and implement them into stretchable electronic devices. In this Account, we report on how the key physical and chemical properties of Ga-based LMs can be leveraged to enable repeatable manufacturing and precise patterning of stretchable LM conductors. A comprehensive understanding of the interplay between the LM, its receiving substrate chemistry and topography, and the environmental conditions is necessary to meet the reproducibility and reliability standards for large scale deployment in next-generation wearable systems. In oxidative environments, a solid oxide skin forms at the surface of the LM and provides enough stiffness to counter balance surface tension, and prevent the LM from beading up to a spherical shape. We review techniques that advantageously harness the oxide skin to form metastable structures such as spraying, 3D printing, or channel injection. Next, we explore how controlling the environmental condition prevents the formation or removes the oxide skin, thereby allowing for selective wetting of Ga lyophilic surfaces. Representative examples include selective plating and physical vapor deposition. The wettability of LMs can be further tuned by engineering the surface chemistry and topology of the receiving substrate to form superlyophobic or superlyophilic surfaces. In particular, our group developed Ga-superlyophilic substrates by engineering the surface of silicone rubber with microstructures and a gold coating layer. Thermal evaporation of Ga on such engineered substrates allows for the formation of smooth LMfilms with micrometric thickness control and design freedom. The versatility of the available deposition techniques facilitates the implementation of LM conductors in a wide variety of wearable devices. We review various epidermal electronic systems using LM conductors as interconnects to carry power and information, transducers and sensors, antennas, and complex hybrid (soft-rigid) electronic circuits. In addition, we highlight the limitations and challenges inherent to the use of Ga LM conductors that include electromigration, corrosion, solidification, and biocompatibility.

Reference

 

Posted on: May 9, 2019