Virtually Transparent Epidermal Imagery

Abstract:

In the process of developing a cyber-physical system capable of displaying the in vivo surgical area directly onto patients' skin, an important research challenge emerged. To generate virtual views from an arbitrary angle, 3D information of internal organ surfaces is crucial. The 3D reconstruction of internal organ surfaces for minimally invasive surgery (MIS) with stereo cameras is usually very difficult due to the challenges in correspondence matching, since there is very limited texture but significant specular reflection on organ surfaces. We observed that the inherent geometric property of organ surfaces could be well captured by parametric models such as thin plate spline (TPS) models. For example, a recent work has applied TPS based 2D tracking on stereo image store cover accurate 3D heart surfaces. Therefore, we use stereo cameras to capture an approximated 3D surface and use it to assist feature matching between images from stereo cameras with different view angles. Furthermore, we have developed a vision based localization and mapping techniques that can simultaneously track the wireless cameras and endoscope and recover a sparse 3D structure of the tissue. To deal with the general tissue deformation, a new framework has been developed with the ability of simultaneous stereoscope tracking, 3D reconstruction and deforming point detection in the MIS environment. First, we adopt a parallel tracking and mapping (PTAM) framework and extend it for the use of stereoscope in MIS. Second, this newly extended framework enables the detection of deforming points without restricted periodic motion model assumptions. The developed method has been evaluated on a phantom model and with vivo data to demonstrate its capability for accurate tracking in nearly real time speed as well as dense 3D reconstruction with hundreds of 3D points. Those experiments have shown that our method is robust in regards to tissue deformation and hence has promising potential for information integration by registration with pre-operative radiologic data such as CT scans.In addition, to transmit high-definition video from multiple wireless cameras inside human abdomen to a receiver outside of the body, we have designed a potential wireless communication scheme. For our system, the typical optimized metrics in communication schemes, such as power and data rate, are far less important than latency and hardware footprint that absolutely preclude their use if not satisfied. We propose the use of Frequency-Modulated Voltage-Division Multiplexing (FM-VDM) where sensor data is kept analog and transmitted via "voltage- multiplexed" signals that are also frequency-modulated. In this manner the overhead required for digitalization and compression of high-definition (HD) video is removed allowing for a wireless link, which may require higher power to overcome noise, but is possible to devise with a tiny footprint and zero-latency video. For education purpose, we have developed and prototyped a low cost Spatial Augmented Game Environment (SAGE) platform based on our main research techniques. Our current system can superimpose full internal organs, muscles, and skeletons on users. The heart and lung are animated with beating and breathing motion respectively. We also animated lungs with illness and the animation is triggered by a smoking gesture. It is widely believed that increased engagement and spatial interaction may provide benefit to learning. To explore this notion, we have tested an experimental Science, Technology, Engineering, and Mathematics (STEM) educational game entitled 'Augmented Anatomy' designed for our proposed platform on a student population. The results indicate that: a) learning of anatomy on-self does reject the null hypothesis and appears correlated with increased engagement and b) a SAGE can be effective at teaching short and long term identification of anatomical structures.