Multimodal Endoscopic Imaging System

We developed a forward-looking (FL) multimodal endoscopic system that offers color, spectral classified, high-frequency ultrasound (HFUS) B-mode, and integrated backscattering coefficient (IBC) images for tumor detection in situ. Examination of tumor distributions from the surface of the colon to deeper inside is essential for determining a treatment plan of cancer. For example, the submucosal invasion depth of tumors in addition to the tumor distributions on the colon surface is used as an indicator of whether the endoscopic dissection would be operated.

Thus, we devised the FL multimodal endoscopic system to offer information on the tumor distribution from the surface to deep tissue with high accuracy. This system was evaluated with bilayer gelatin phantoms which have different properties at each layer of the phantom in a lateral direction. After evaluating the system with phantoms, it was employed to characterize forty human colon tissues excised from cancer patients. The proposed system could allow us to obtain highly resolved chemical, anatomical, and macro-molecular information on excised colon tissues including tumors, thus enhancing the detection of tumor distributions from the surface to deep tissue. These results suggest that the FL multimodal endoscopic system could be an innovative screening instrument for quantitative tumor characterization.

Multimodality of an optical system implies the use of one or more optical techniques to improve the system's overall performance and maximum utility. We demonstrate a multimodal system with oblique illumination that combines two different techniques; fluorescence micro-endoscopy and spectroscopy simultaneously and can be utilized to obtain diverse information from the same location of biological sample. In present system, use of graded index (GRIN) rod-lens makes it highly compact and oblique incidence decouples illumination geometry with collection geometry, preventing CCD cameras from saturation and reduces number of optical elements, thereby making system further miniaturized and field-portable. It also overcomes the disadvantages of undesired reflections from different optical elements.

The experimental results of simultaneous imaging and spectroscopy of the biological samples are presented along with quantitative spectroscopic parameters; peak wavelength shift, area under the curve and full width half maximum (FWHM). Furthermore, we have obtained the red shift for cancerous oral tissue with respect to normal oral tissue 5.79 ± 1.071 nm. This could be important indicator for oral cancer screening.

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework.

To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.