With the rapid development of sensor networks, machine vision faces the problem of storing and computing massive data. The human visual system has a very efficient information sense and computation ability, which has enlightening significance for solving the above problems in machine vision. This review aims to comprehensively summarize the latest advances in bio-inspired image sensors that can be used to improve machine-vision processing efficiency. After briefly introducing the research background, the relevant mechanisms of visual information processing in human visual systems are briefly discussed, including layer-by-layer processing, sparse coding, and neural adaptation. Subsequently, the cases and performance of image sensors corresponding to various bio-inspired mechanisms are introduced. Finally, the challenges and perspectives of implementing bio-inspired image sensors for efficient machine vision are discussed.
Transmission matrix (TM) has applications ranging from imaging through scattering, communication, to multimode fiber imaging. TM retrieval for binary amplitude modulation recovers the TM from the intensity outputs of probing binary incident fields. However, the computational complexity limits the application for retrieval of large TM. We propose an efficient algorithm for TM retrieval with binary amplitude modulation. Our method designs the probing binary fields with convolution matrix and develops efficient retrieval algorithm based on fast Fourier transform (FFT). It improves the computational complexity by orders of magnitude. We verify the proposed algorithm with simulated data.
High performance multimode fiber (MMF) endoscopy is a possible thinnest endoscopy for widely applications, The main problem is how to real time overcome the mode coupling during the movement of the fiber and to get the high performance imaging. In this paper, we present our method to high speed compensating the mode coupling during the fiber movements, and present the possibility of the multimode fiber can not only be the finest endoscopy imaging, but also can be used to realize multi-modal imaging.
KEYWORDS: In vivo imaging, Super resolution, Nanoimaging, Image resolution, Microspheres, Evanescence, Spatial frequencies, Multimode fibers, Modulation, Imaging systems
Existing super-resolution imaging technology relies on rigid and bulky systems, which limits its application in narrow space. Endoscopy based on multimode fibers (MMF) has emerged as a significant solution for in vivo imaging. There is a high requirement to observe samples inside the lumen of the body simultaneously break through the diffraction limit. We propose the frequency-shift mechanism for universal super-resolution imaging with light-field encoded modulation. In this paper, we demonstrate our method for a single MMF in vivo imaging at subcellular resolution through light-field encoded, which can also ensure stable imaging under various operating conditions.
Image projection through a multimode fiber (MMF) or scattering media has applications ranging from optogenetics to near eye-displays. It requires developing computer-generated holography algorithm to obtain phase pattern of spatial light modulator. In order to accurately project light, conventional methods measured the transmission matrix (TM) of the imaging system by interference. However, it is sensitive to phase instability, easily caused by thermal drift and mechanical vibration. In this work, we proposed to use the TM retrieved from intensity-only measurements and develop a nonlinear optimization algorithm to obtain the displayed phase patterns. Our method formulates the forward model with the retrieved TM, derives the analytical derivative and adopts a second-order optimization method. We validate the improved quality of the projected intensity image by an experiment setup with a MMF.
Frequency-shift super-resolution (SR) microscopy, such as structured illumination microscopy (SIM) and Fourier ptychographic microscopy (FPM), can break the diffraction limit for the imaging of both fluorescently labeled and label-free samples by transferring the high spatial-frequency information into the passband of microscope. However, conventional SIM microscopy systems tend to be bulky and expensive, which limits its applications in various fields. Therefore, we’ve developed some chip-based frequency-shift SR technologies, which is compatible with conventional microscope, and can be further designed for portable imaging systems such as smart phone and so on. We also developed a deep-learning based imaging method to improve the imaging speed of frequency-shift super-resolution microscopy, which enables low-cost and fast super-resolution imaging for real-time live cell biological studies.
KEYWORDS: 3D image reconstruction, Natural surfaces, 3D image processing, Tissues, Polarization, Polarimetry, Depth maps, 3D modeling, 3D acquisition, Surgery
By acquiring three-dimensional profiles of biological tissues, interventions can be performed with increased speed and accuracy, driving the development of next-generation image guided therapy. However, current three-dimensional reconstruction techniques relying on feature detection and matching struggle with tissues lacking distinct features, resulting in relatively sparse reconstruction results. In this paper, we propose a data-driven method for reconstructing three-dimensional surfaces from a single polarimetric image, utilizing physics-based priors. We constructed a calibrated imaging system consisting of a polarization camera and a 3D scanner to collect polarization information and ground truth 3D data. Using this system, we created a dataset with organ models, capturing polarization images, depth maps, and surface normal maps under different lighting conditions. To achieve our goal, we designed a deep neural network based on the Unet architecture. This network takes the polarization image and prior physical parameter maps (phase angle, degree of polarization, and unpolarized intensity) as inputs and is trained to output the surface normal map and relative depth map of the organ. Experimental results on the tissue phantom dataset demonstrate the effectiveness of our method in generating dense reconstruction results, even for the regions lacking distinct features. Furthermore, we validated the robustness of our method to changes in the light source direction, showcasing its ability to handle variations in lighting conditions. Overall, our proposed data-driven approach provides a promising solution for dense three-dimensional reconstruction from a single polarimetric image, leveraging physics-based priors and deep learning techniques.
Multispectral endoscopic imaging is a promising technique for lesion detection, and surgical guidance. Based on different spectral properties of tissues, multispectral imaging can provide enhanced contrast of vascular structures (narrow band imaging) or enable the quantitative analysis of hemoglobin and lipid. The multiparameter phantoms serve as efficient tools for system calibration, performance evaluation, and algorithm development in multispectral endoscopy are needed. In this paper, we developed a multiparameter tissue-mimicking phantom that mimics the parameters of human gastric mucosa, such as scattering coefficient, scattering layer thickness, vascular width, lipid on the surface and blood oxygen saturation (SO2). We verified the SO2 measurement accuracy by comparing with commercial i-STAT devices for SO2 distribution imaging. At the same time, the segmentation of lipid regions was also tested. Our results demonstrate that this multiparameter phantom is a versatile tool that can facilitate validation and evaluation of multispectral endoscopic systems.
Structured light 3D surface imaging reconstructs the 3D surface shape of an object by analyzing the deformation of a projected structured-light pattern. The detection of the pattern is a crucial step in this technique, which can be affected by the blurring of the patterns due to tissue scattering and absorbing. In this study we experimentally investigate how the projected structured light based on hyperspectral projection. A hyperspectral projector is constructed to generate patterns of different wavelengths, bandwidth and densities. The projected structured light is tested on porcine tissue in vitro. The sharpness of all the patterns and property of tissue are evaluated. The results indicate that tissue-dependent wavelength of light with appropriate density of pattern yield optimal illumination which corresponds to 3D reconstruction accuracy for structured light 3D endoscopy.
Imaging through multimode fiber (MMF) provides high-resolution imaging through a fiber with cross section down to tens of micrometers. It requires interferometry to measure the full transmission matrix (TM), leading to the drawbacks of complicated experimental setup and phase instability. Reference-less TM retrieval is a promising robust solution that avoids interferometry, since it recovers the TM from intensity-only measurements. However, the long computational time and failure of 3D focusing still limit its application in MMF imaging. We propose an efficient reference-less TM retrieval method by developing a nonlinear optimization algorithm based on fast Fourier transform (FFT). Furthermore, we develop an algorithm to correct the phase offset error of retrieved TM using defocused intensity images and hence achieve 3D focusing. The proposed method is validated by both simulations and experiments. The FFT-based TM retrieval algorithm achieves orders of magnitude of speedup in computational time and recovers 2286 × 8192 TM of a 0.22 NA and 50 μm diameter MMF with 112.9 s by a computer of 32 CPU cores. With the advantages of efficiency and correction of phase offset, our method paves the way for the application of reference-less TM retrieval in not only MMF imaging but also broader applications requiring TM calibration.
In recent years, endoscopic imaging has become the major diagnostic approach for early cancer. It is difficult to accurately obtain the information of lesions and components content by using conventional white light imaging (C-WLI). Multispectral imaging techniques such as narrowband imaging are widely used in endoscopic clinical because of their specificity on the surface structure of digestive tract mucosa. However, there is still a lack of multispectral techniques for tissue components specificity. Tissue components such as lipid, hemoglobin are closely related to the generation and development of tumors, but it is difficult to observe the characteristics of lipid and hemoglobin by conventional white light imaging. Therefore, we studied the multispectral endoscopic imaging technique for the analysis of digestive tract mucosal components. Based on the reflectance spectrum characteristics of these tissue components, we determined their specific wavelength, and comprehensively considered the absorption, scattering, oxygenation and others in order to determine the most appropriate optical band in the wavelength range of 365-660nm. Through experimental verification, we choose different narrow-band wavelength combinations for lipids and hemoglobin oxygen saturation respectively to achieve multispectral imaging. In order to simulate the optical properties of digestive tract surface, we made tissue optical phantom. At the same time, we tested and optimized the imaging system and algorithm by the experiments in vitro, and obtained the optimal multispectral image of tissue components, then realize the quantitative detection combined with the content analysis algorithm. Our pilots show that the multispectral imaging system can improve the contrast of endoscopy image, enhance the detail information, achieve high precision detection of tissue components content, and control the error within 10%.
Polarimetric imaging can provide morphological and structural information of tissue and has demonstrated applications during endoscopy. Polarimetric endoscopy has been shown by conducting polarimetric imaging through customized rigid endoscopes. Here we demonstrate Stokes polarimetric imaging via a commercially available rigid endoscope that can acquire full-Stokes information in a snapshot. This system incorporates two division of focal plane linear polarization cameras, a waveplate and a commercially available rigid endoscope. In addition, the sapphire windows in rigid endoscopes are known for their polarization effects. In this work, the polarization properties of the sapphire windows are carefully measured and calibrated to guarantee the accuracy of the system. Quantified analysis of the system and the imaging results on tissue are provided.
The main challenge in multimode fiber imaging is modal scrambling caused by environmental fluctuation. How to get high contrast and high stable imaging is the main question. In this presentation, we propose some methods to increase the contrast-to-noise ratio and stability of multimode fiber imaging. Wavelength modulation is introduced to suppress the background. Exhaustive bending effect was used to improve the imaging stability. Wavelength modulation is introduced to enhance the CNR four fold in a 200 μm field-of-view imaging. We show a near diffraction limited focusing capability at imaging depths of up to 150 µm with near constant lateral resolutions of 2.1 µm. The imaging of small fluorescent beads embedded in a 3D matrix was demonstrated.
Linear super-resolution microscopy via synthesis aperture approach permits fast acquisition owing to its wide-field implementations. However, it has been limited in resolution because a spatial-frequency band missing occurs when trying to use a shift magnitude surpassing the cutoff frequency of the detection system beyond a factor of two, which distorts the image severely. Here, we propose a method of chip-based 3D nanoscopy through a tunable spatial-frequency-shift effect capable of covering the full extent of the spatial-frequency component within a wide passband. The missing of the spatial spectrum can be effectively solved by developing a spatial-frequency-shift active tuning approach through wave vector manipulation and operation of optical modes propagating along multiple azimuthal directions on a waveguide chip. Besides, the method includes a chip-based sectioning capability, which is enabled by the saturated absorption of fluorophores.
Optics and light-based technology are becoming more and more important as a cross-field of new science and engineering, and the scope of optics education should be extended. Since 2018, in Zhejiang University a new wave of general education reform has started, and our course Light-based Science and Technologies and Human Civilization has become a common core course. In the last 3 years our students mainly come from science and engineering, only a few students come from social science. From now on we will offer this course to multidisciplinary students, especially the students of politics, economics, law and arts. In this paper we present our learning organization, especially the group discussion in multidisciplinary students. The students from different specialties are divided into one group. The discussion topics are about research methodology, famous arguments in optics history, optics in daily life and imagination of the future.
Far-field super-resolution imaging techniques such as stimulated emission depletion (STED), stochastic optical reconstruction microscopy (STORM), and photoactivation localization microscopy (PALM) etc., have enabled fluorescence labeled nanoscale imaging and analysis. However, it is necessary or preferable to develop label-free high resolution imaging techniques to avoid the problems of phototoxicity with live cells and moreover, for many non-biological applications where fluorescent labelling is simply not feasible. Recently, by capturing the exponentially decaying evanescent waves and propagating it to far field, novel imaging technique like hyperlens and microsphere contacting technologies, micro-fiber and nanowire illumination technology have brought new opportunities to label-free super resolution imaging, but still suffer from weak signal compared to background noise. On the other hand, there is lack of criteria reported to quantify the imaging quality of label free far-field nanoscopy, that will slow down the developing of label-free far-field nanoscopy. We introduce CNR first to label-free far-field nanoscopy to quantify the imaging performance and investigate the key influencing elements systematically. In our study, we investigated and analyzed the key elements to achieve high CNR label-free wide-field far-field sub-diffraction imaging. By optimizing the key factors including polarization, materials, as well as fabrication conditions, sub-diffraction imaging with resolution of 122 nm in a large FOV has been achieved experimentally. This work has provided an efficient and convenient way to realize high contrast wide-field far-field label-free super-resolution based on evanescent wave illumination.
KEYWORDS: Light sources, Spatial frequencies, Luminescence, Nanoimaging, Signal to noise ratio, Light, Waveguides, Nanostructures, Evanescent wave coupling, Digital video discs
We demonstrate a novel active far-field wide-field label-free nano-imaging with high signal-to-noise ratio based on the nanoscale light source, pioneering in utilizing a combination of spatial frequency shift and Stokes frequency shift. Ring shape or polygonal shape nanoscale light sources emit photoluminescence and the light is coupled to a underneath subwavelength film waveguide to produce omnidirectional illuminating evanescent waves to shift high spatial frequencies for the far-field, label-free, 2D subdiffraction imaging. The photoluminescence of the nanostructure enables high-efficiency evanescent wave coupling in a large area for the low-noise wide-field imaging. The field of view is one order of magnitude improved from the previously reported far-field and full-field method. This approach’s versatility has been demonstrated by imaging integrated chips, Blu-ray DVDs, biological cells and various subwavelength 2D patterns. The configuration-----a special NW assisted slide-----is cost-effective, quite compatible with the standard microscope and can conveniently give super resolvability to conventional microscopy as a compact module. This work is significant for physics, materials, biology, chemistry, and other areas that need nanoscale visualization.
Spectroscopy has a long history. The theory of is difficult for students to understand. So we want to improve the traditional teaching to some way of interesting experience combined with historical knowledge, practical application and development frontiers. We make use of all kinds of resources to get vivid information of big events of spectroscopy development in order to show students the specific process of some phenomenon. Meanwhile, students will be suggested to read all kinds of latest papers relevantly to obtain much more information about this discipline. Both in class and in lab, we lead students to do some very useful experiments and give them guidance. Through this practice, they will understand the theory much more deeply, especially they will know how to solve the problems in research.
Starting from 2015, a general education course named “Light-based science and technologies and human civilization” has been offered in Zhejiang University. We try to give a humanism view angle to observe optics and optical engineering, and combine them with the relationship of human and the nature, the development of human society and human health. In this course we introduce different historical periods of light-based science and technologies, the great optical researchers, the typical research methods, advantages, academic discussions and the relationship with human civilization. The relevant cross-fields of learning and Nobel Prize winners are also included. This course provides the students with the typical examples about how academic revolution influences the world development, and also with humanism sight which exceeds the range of science and technologies themselves.
Nanoscale lasers are the key component in the integrated photonics chips and have attracted much interests. Nanoblets and nanowires lasers, as one of the candidates for the nanoscale lasers, have been developed for one more decades. Many kinds of nanowire lasers with different functionalities, such as wavelength tunable, single mode, polarized emission and so on, have been demonstrated. However, the reported single mode nanowire lasers are mostly realized through microfabrication process, careful manipulation and complicated structures. Here, we present a new type of lateral emission single mode nanobelt lasers with high polarization ratio which are fabricated by the one step traditional VLS (Vapor Liquid Solid) growth. Different from the traditional nanobelt lasers which are based on the FP cavity formed in the longitudinal direction, the emission of this novel nanoblet laser is lateral which is contribute to the special wire-like structures grown on the nanobelt. It shows band edge emission and the wavelength is centered at 712.6 nm with linewidth about 0.18 nm. The threshold reach as low as 15 uJ/cm2 benefit from the unique morphology which provides enhanced confinement factor for optical modes. Meanwhile the laser emission is highly polarized with polarization ration as high as 0.91. This lateral emission single mode nanobelt laser with high polarization ratio, low threshold and simple fabrication technique offers an economic and effective choice to the future optical applications.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.