2.1.

Metalens Imaging System

Metalenses are fabricated through nanoimprint lithography and subsequent atomic layer deposition.¹³ Nanoimprint lithography provides the benefits of low-cost mass production and uniformity of the products.^7,8,13,40 Thus, we use imprinted metalenses to broadly impact our work on the commercialization of the deep neural network (DNN)-based metalens imaging system. Figure 2(a) shows mass-produced 10-mm-diameter metalenses fabricated by nanoimprint lithography and subsequent thin-film deposition of ${\mathrm{TiO}}_2$; the details of the fabrication process are given in Supplementary Material. As shown in Fig. 2(b), our metalens comprises nano-slabs with arbitrary rotational angles as a PB-phase-based metalens. It has a relatively high efficiency of 55.6% at 532 nm wavelength but exhibits severe chromatic aberration. As shown in Fig. 2(c), the focal lengths at wavelengths of 450, 532, and 635 nm are 29.0, 24.5, and 20.5 mm, respectively.¹³ This wavelength-dependent focal length results in transverse axial chromatic aberration (TAC), which can be expressed as

Fig. 2

(a) Photograph of fabricated mass-produced 10-mm-diameter metalenses on 4″ glass wafer. The inset in the red box shows enlarged image of the metalens. (b) Scanning electron microscopy (SEM) image showing the meta-atoms that compose the metalens. The scale bar is $3\mu \mathrm{m}$. (c) Focal lengths of the metalens for wavelengths of 450 nm (blue), 532 nm (green), and 635 nm (red). The dashed line indicates the linear fitting result. (d) MTFs of red, green, and blue lights with zero viewing angle. (e) PSFs of red, green, and blue lights with various viewing angles (0 deg, 5 deg, 10 deg). The scale bar is 1 mm, which indicates a distance on the image sensor. (f) Metalens image (left) and its subset images showing red, green, and blue color channels. (g) Corresponding ground truth image (left) and its subset images showing red, green, and blue color channels.

The metalens imaging system is affected by chromatic and angular aberrations and its surface defects by incomplete fabrication. To quantify these effects, we measured the PSFs and calculated the modulation transfer function (MTF) from the measured PSFs. The PSF, which is the two-dimensional (2D) intensity distribution obtained in response to a single point light source,⁴¹ is a critical metric for evaluating the quality of an imaging system because it is directly related to image formation.⁴² The MTF, calculated using the measured PSFs, describes the imaging quality in terms of resolution and contrast.⁴¹ We measured the PSFs by capturing the images of collimated beams from red, green, and blue light-emitting diodes (LEDs) using the metalens imaging system and subsequently calculated the MTFs from the PSFs. The PSF measurement and the imaging setup for it are shown in Fig. S2 in the Supplementary Material and elaborated upon, and the MTF calculation method is also subsequently explained in detail in the Supplementary Material.

Figure 2(e) shows the PSFs of red, green, and blue LEDs $i$’th various viewing angles (0 deg, 5 deg, 10 deg). The PSF profiles of red and green LEDs show wide disk shapes, whereas the profile of the green LED shows irregular spark shape, implying the effect of the TAC. Thus, as shown in Fig. 2(d), the MTFs of the red and blue lights are severely lower than the MTF of the green light at all spatial frequencies. Furthermore, the PSF profiles at 0 deg viewing angle show non-ideal and circularly asymmetric shapes, which can be attributed to the defects of the metalens due to the imperfect fabrication. The PSF profiles also change their shapes with the viewing angle due to the angular aberrations, including Seidel aberrations³ and the angular dispersion of the meta-atoms.²⁸ The PSF profiles of the red and green LEDs stretch to the horizontal direction as the viewing angle increases, where the profile of the blue light shrinks. In addition, the non-uniformity of the metasurface during the fabrication process¹³ may result in the PSFs with complex profiles not matching the PSFs from the Rayleigh-Sommerfeld diffraction formula.⁴³ As a result, the combination of these generates complex PSF profiles varying with the viewing angle and further complicates the image restoration tasks.

The effects of chromatic and angular aberrations on the metalens images can be shown by comparing them against the ground truth image. Figures 2(f) and 2(g) show the metalens image, the corresponding ground truth image, and the subset images depicting the red, green, and blue color channels. The red and blue channels of the metalens image are severely blurred from the TAC, making it difficult to recognize any objects. In addition, unlike the PSF measurements, the blue color channel appears more blurry than the red color channel due to the optical setup for data acquisitions as shown in Fig. S1 in the Supplementary Material. The green channel of the metalens image shows a relatively higher resolution at the center, which gradually decreases as the viewing angle increases (e.g., the outer region of the image) due to the angular aberrations at the higher viewing angle.

2.2.

Image Restoration Network

Computational image restorations have emerged as a prevalent approach for the enhancement of non-ideal images, such as those that are noisy⁴⁴ or blurred.⁴⁵ Classical image restoration methods achieve higher resolution by relying on linear deconvolution methods, such as applying the Wiener filter.⁴⁶ Deconvolution, an inverse of the convolution operation, facilitates the recovery of the original image from an image convolved with a PSF. The performance of the deconvolution process depends on two factors: the space invariance of PSF across the FoV and the low condition number for the inverse of the PSF.⁴⁷ However, Wiener filters exhibit limited restoration quality for imaging systems with PSFs that vary depending on the viewing angle, such as metalens imaging systems³⁶ and under-display cameras.⁴²

An alternative restoration approach is the utilization of DNN-based image restoration. DNN-based restoration models^38,39 have shown superior performance compared to traditional approaches in specialized tasks, such as denoising,⁴⁴ de-blurring,⁴⁵ super-resolution,⁴⁸ and light-enhancement.⁴⁹ Furthermore, they are applicable to imaging systems with complex and combined degradations, such as under-display cameras⁴² and the 360 deg FoV panoramic camera.⁵⁰ However, conventional DNN approaches are incapable of learning position-variant image degradations (e.g., position-dependent aberration of the metalens) because these methods train models with randomly cropped patches from full-resolution images, leading to the complete loss of position-dependent information.

In response to these challenges, we propose an end-to-end image restoration framework specifically tailored for the metalens imaging system to address non-uniform aberration over the wavelength and viewing angle. Contrary to the images that are subjected to restoration in typical image restoration tasks,^51,52 our metalens images exhibit more intense blur and significant color distortion. Consequently, the restoration of metalens images constitutes a severely ill-posed inverse problem. To address this critically underconstrained problem, we employ strong regularization. That is, we model the traits and patterns of sharp data, performing adversarial learning in Fourier space to train the data distribution. Therefore, the restoration model $f(y)$ is trained by minimizing

2.2.1.

Network architecture

The architecture of our image restoration framework is depicted in Fig. 3. Our framework incorporates existing DNN architecture with our proposed methods. The training phase involved the utilization of patches randomly cropped from images at their full resolution, specifically $1280\times 800$ in this study. Subsequently, in the inference phase, the analysis was conducted on the entire images at their original resolution of $1280\times 800$. However, we observed a statistical disagreement in the inference process of the full-resolution image, as shown in Fig. S3 in the Supplementary Material. To overcome this inconsistency, we apply test-time local converter (TLC)⁵³ in the test phase, which yields a significant performance improvement. The detailed results are presented in Table S1 in the Supplementary Material.

Fig. 3

Proposed image restoration framework. The framework consists of an image restoration model and applies random cropping and position embedding to the input data using coordinate information of the cropped patches. To address the underconstrained problem of restoring degraded images to latent sharp images, adversarial learning in the frequency domain is applied through the FFT ($\mathcal{F}$). $\widehat{x}$ and $x$ denote the reconstructed and ground truth image, respectively. The details of the framework are in Sec. 2.

The metalens used in our study exhibits intense chromatic and angular aberrations, resulting in severe information loss in the images captured with it. Therefore, we trained the model according to the traits and patterns found in the underlying clean images to efficiently restore a wide range of spatial-frequencies and constrain the space of the latent ground truth images. Because generative models can learn complex, high-dimensional data distributions from a given dataset,⁵⁴ we utilized an adversarial learning scheme, one of the generative learning methods, to learn effectively the distribution of latent sharp images by introducing an auxiliary discriminator. We initially applied adversarial learning in the RGB space but observed that conspicuous pattern artifacts appeared in both the RGB and Fourier spaces (Fig. S4 in the Supplementary Material). These artifacts, related to periodic patterns, are more clearly visible in the Fourier domain than in the RGB space due to their deep connection with spectral components [Figs. S4(c) and S4(d) in the Supplementary Material]. Since the Fourier space provides a more explicit representation of these spectral components, it allowed us to identify better and address the source of the artifacts. Therefore, we transformed the data from each RGB channel into the Fourier space for adversarial learning. These Fourier space data are then used as input for the discriminator.

The training loss is composed of two distinct terms: peak signal-to-noise ratio (PSNR) loss $L_{\mathrm{PSNR}}$ between the reconstructed image $\widehat{x}$ and the ground truth image $x$ and adversarial loss $L_a$ between $\mathcal{F}(\widehat{x})$ and $\mathcal{F}(x)$. The PSNR loss signifies the image fidelity loss, and the adversarial loss indicates the prior regularization loss. Therefore, the total loss function $L_{\text{Total}}$ is

Eq. (3)

$$L_{\text{Total}}=L_{\mathrm{PSNR}}+\lambda L_a,$$

where $\lambda$ is a hyperparameter for balancing $L_{\mathrm{PSNR}}$ and $L_a$. $L_{\mathrm{PSNR}}$ is calculated as follows:

Eq. (4)

$$L_{\mathrm{PSNR}}(\widehat{x},x)=-10\log \frac{R^2}{\mathrm{MSE}(\widehat{x},x)},$$

where $\widehat{x},x$, and $R$ denote the reconstructed image, ground truth image, and the maximum signal value of the ground truth image, respectively. MSE is the distance between the reconstructed and ground truth images and is formulated as $\mathrm{MSE}(\widehat{x},x)=\frac{1}{N}\sum _{n=1}^N({\widehat{x}}_n-x_n{)}^2$.

For adversarial learning, we constructed an additional discriminator and applied spectral normalization⁵⁵ for training stability. In addition, we employ the GAN training scheme based on hinge loss⁵⁶ for enhanced stability of adversarial training. The adversarial loss $L_a$ of the discriminator ($\mathrm{D}$) and the image restoration model ($\mathrm{G}$) is

Eq. (5)

$$L_a^{\mathrm{D}}={\mathbb{E}}_x[\max (\mathrm{0,1}-D(\mathcal{F}(x)))]+{\mathbb{E}}_{\widehat{x}}[\max (\mathrm{0,1}+D(\mathcal{F}(\widehat{x})))],$$

Eq. (6)

$$L_a^{\mathrm{G}}=-{\mathbb{E}}_{\widehat{x}}[D(\mathcal{F}(\widehat{x}))],$$

where $\mathcal{F}$ refers to fast Fourier transform (FFT). Here, ${\mathbb{E}}_x[·]$ and ${\mathbb{E}}_{\widehat{x}}[·]$ are operators that denote the calculation of the mean of the ground truth and reconstructed images in the given minibatch, respectively. The image restoration model and discriminator each try to minimize $L_a^{\mathrm{G}}$ and $L_a^{\mathrm{D}}$, respectively.

Degradation in the outer region of the metalens image is more pronounced than in the central region due to the angular aberration. This observation suggests that positional information is integral for understanding the degradation of the metalens imaging system. However, the training method makes it impossible for the model to learn positional information because our framework learns through random patches during training and restores full-resolution images during inference.

To address this problem, we take the coordinate values of each pixel of the patches, based on the coordinates of a full resolution image, and map them through a $1\times 1$ convolutional layer. This process transforms them into proper space when generating random patches for a full-resolution image. The processed coordinate information is concatenated with metalens images corresponding to the information. The resulting concatenated data are used as input data. This approach enables the model to learn and leverage positional information effectively, thereby enhancing its performance in restoring full-resolution images.

3.1.

Quality of Image Restoration

Figure 5 comprehensively shows the results of the PSNR, structural similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS) in RGB space, and mean absolute error (MAE) of the magnitudes, as well as cosine similarity (CS) in Fourier space calculated by comparing the metalens image and the image reconstructed by our framework with the ground truth image. The red horizontal lines in each box represent the median, and the boxes extend from the first to the third quartile. The whiskers span 1.5 times the interquartile range of the first and third quartiles. We conducted a statistical hypothesis test to ascertain whether the observed results exhibit statistically significant differences. This was accomplished through the utilization of a two-sided paired $t$-test to evaluate the performance disparity between images produced by metalenses and those reconstructed by the proposed framework. A significance level of $P={10}^{-4}$ was set for the testing process.

Fig. 5

Comparative statistical analysis of the proposed model and metalens imaging results using the test dataset. (a)–(e) Results of PSNR, SSIM, LPIPS in RGB space and CS, MAE of the magnitudes in Fourier space calculated by comparing the metalens image and the image reconstructed by our framework with the ground truth image. A statistical hypothesis test was performed through a two-sided paired $t$-test on the performance difference between the metalens image and the image reconstructed by our framework [significance level $P={10}^{-4}$, (a) $1.055\times {10}^{-39}$, (b) $3.886\times {10}^{-35}$, (c) $1.363\times {10}^{-48}$, (d) $2.311\times {10}^{-35}$, and (e) $2.150\times {10}^{-38}$].

Within this analysis, the outcomes indicate a statistically significant variance across all evaluated metrics, as evidenced in Fig. 5. These metrics were assessed utilizing a test set comprising 70 data points. Also, Table 1 shows the quantitative results of the metalens imaging system, our framework, and state-of-the-art models for various metrics. The implications derived from each graph and the significance of the quantitative outcomes are elaborated below, providing a comprehensive analysis of the data and its relevance to the study’s objectives.

Table 1

Comparison of quantitative assessments of various models using the test set of images (n=70). The first and second values of each column represent the mean and the standard deviation of the metrics, respectively. The best scores are marked as bold.

To further understand the impact of our framework on the fidelity of image restoration, we examine PSNR and SSIM,⁵⁹ which serve as the foundational metrics. The former is a quantitative measure of the restoration quality of an image, calculated as the logarithmic ratio between the maximum possible power of a signal (image) and the power of corrupting noise that affects its fidelity. Higher PSNR values indicate better quality of the reconstructed image. The latter, SSIM, valuates the visual impact of three characteristics of an image: luminance, contrast, and structure, thus providing a more accurate reflection of perceived image quality.

Figure 5 presents a statistical analysis comparing the PSNR and SSIM values of the images captured through the metalens with those restored by our framework. As shown in Table 1, the framework showcased a remarkable improvement in image fidelity, elevating the PSNR by 7.37 dB and SSIM by 22.5%p compared to the original metalens images. These enhancements underscore our framework’s proficiency in mitigating the fidelity loss incurred by metalens aberrations, thus significantly elevating the quality of the reconstructed images closer to their ground truths.

While PSNR and SSIM are advantageous for assessing image fidelity and perceptual quality, they often fall short in evaluating the structured outputs. This limitation stems from their inability to fully capture the human visual system’s sensitivity to various image distortions, particularly in textured or detailed regions. To address this gap, LPIPS⁶⁰ was employed to evaluate the perceptual quality of the images. LPIPS evaluates perceptual similarity by utilizing pretrained deep learning networks (e.g., AlexNet), offering a nuanced measure that aligns more closely with human perception of image quality. Lower LPIPS values indicate better perceptual quality.

Table 1 demonstrates that our framework achieved a 35.6%p decrease in LPIPS, indicating a substantial enhancement in the perceptual resemblance of the reconstructed images to their original counterparts, as also observable in Fig. 5(c). This metric highlights the proposed framework’s capability to not only improve the objective quality of images but also their subjective, perceptual quality. We also compare our framework with state-of-the-art models, including restoration models for natural images (MIRNetv2,⁵⁷ HINet,⁵⁸ NAFNet³⁸). As shown in Table 1, our framework surpasses these state-of-the-art models by a substantial margin in terms of PSNR, SSIM, and LPIPS. In addition, we conduct further experiments to measure and compare the restoration performance for spatially and spectrally varying degradations (Tables S4 and S5 in the Supplementary Material). This suggests that our framework is more suitable for the metalens image restoration task than conventional models designed for restoring natural images, such as those in the DIV2K dataset.⁵¹

The measured MTF of the metalens in Fig. 1(d) and qualitative results in Fig. 4(b) demonstrate intense degradation at high spatial frequencies. Consequently, it is crucial to restore the spatial-frequency information during the metalens image restoration task. It is pertinent to acknowledge that spatial frequency can be represented as both magnitude and phase components, with the latter often heralded as important in signal processing realms.⁶¹ We utilize two metrics to evaluate the magnitude and phase of the Fourier-transformed reconstructed images. In evaluating the fidelity of the reconstructed images, particularly concerning their frequency-dependent attributes, two metrics are employed: the MAE for assessing discrepancies in magnitude relative to the original images, and the CS for gauging phase congruence with the authentic images. These metrics are derived through the application of the FFT across images revitalized by disparate models. The ensuing MAE and CS metrics underscore a remarkable enhancement in image quality, as elucidated in Figs. 5(d) and 5(e) and Table 1. As shown in these figures, our framework demonstrates the superior performance of MAE and CS to the metalens imaging system and several state-of-the-art image restoration models in the frequency domain. Our framework achieved about twice the performance of the metalens imaging system for MAE and accomplished about 14%p for CS for the metalens images.

To demonstrate the restoration of the blur and color distortion visually, we tested our imaging system using 1951 U.S. Air Force resolution test chart images (USAF images). Figures 6(a) and 6(b) show monochromatic white and black USAF images captured by the metalens imaging system. These images exhibit severe blurring and strong color distortion, particularly showing greenish tints in white patterns. As shown in Figs. 6(c) and 6(d), the restored images illustrate that the pattern’s colors are closer to white and black than the metalens images. Furthermore, the central regions of the images exhibit high sharpness, while the damaged images have severe blurring in these areas. Thus, our framework demonstrates superior prominence in enhancing overall image quality by achieving conspicuous color fidelity and sharpness.

Fig. 6

(a) and (b) White and black USAF images captured by the metalens imaging system, respectively. (c) and (d) White and black USAF images restored by our framework, respectively. The image in the red boxes shows the enlarged image in the central region indicated as red box. The scale bars in the original and enlarged images are 3 and 0.5 mm, respectively, indicating the distance on the image sensor.

3.2.

Object Detection Performance

We also assess the integrated system’s utility beyond image quality enhancement by transitioning to one of the domains of practical applications, object detection. To validate the performance of our framework in object detection on the restored images, we first obtained a test dataset that consists of the ground truth, metalens, and restored images using the entire PASCAL VOC2007 dataset ($n=4952$).⁶² This dataset comprises 4952 images with the information of the positions of objects and bounding box annotations for instances belonging to 20 different categories. Then, we employ a single shot multibox detector (SSD)⁶³ pre-trained for the PASCAL VOC2007 to detect bounding boxes of given images. Especially, we introduce mean average precision (AP) to evaluate object detection results. AP measures the model’s accuracy in predicting the presence and correct localization of objects within an image. It provides a comprehensive assessment of the detection performance across varying thresholds of precision and recall, making it a standard benchmark in evaluating object detection algorithms. Specifically, AP is calculated in multiple scales using IoU thresholds ranging from 0.5 to 0.95. The terms “${\mathrm{AP}}_{50}$” and “${\mathrm{AP}}_{75}$” each represent the results of measurements with the IoU threshold set to 0.5 and 0.75, respectively.

Figure 7 shows examples of object detection using SSD. The detector predicts the entire region (red box) as an object because it cannot identify the detailed features in the metalens images [Fig. 6(b)]. On the other hand, the detector accurately predicts the bounding boxes at the desired objects in the restored images because compared to the original PASCAL VOC2007, the quality of the restored images is competitive [Figs. 6(a), 6(d) and 6(c), 6(f)]. Our $\mathrm{AP},{\mathrm{AP}}_{50}$ on the restored images are $\sim 34\%\mathrm{p}$ and 56%p higher than those on the metalens images. They are $\sim 86\%$ and 88% of the AP, ${\mathrm{AP}}_{50}$ on the ground truth (Table S2 in the Supplementary Material). The improvement in AP scores for our framework-restored images compared to the original metalens images signifies the restoration’s practical implications. Higher AP scores on our framework-restored images indicate that the model effectively recovers enough detail and structure from the aberrated metalens images to facilitate accurate object detection, closely approximating the performance on ground truth images. This enhancement is particularly crucial for applications in autonomous navigation, surveillance, and augmented reality, where precise object detection is paramount.

Fig. 7

Object detection results using a pre-trained SSD model on (a), (d) the original images, (b), (e) the metalens images, and (c), (f) the images restored by our framework. The pre-trained SSD model could not detect any objects in the metalens images accurately; however, it successfully captured multiple classes and objects in images restored by our framework.