Graphs in pattern recognition have been studied since the late seventies with alternating events of successes and failures. Far from wanting to propose a unique survey on the subject, we aim to present the areas in which graphs have been shown to be effective and to illustrate the problems still open in this research domain, with a focus on recent developments moving in the direction of deep learning. This work intends to arouse the interest of neophytes to this interesting domain that is the representation of the world through graphs.

Facing the black-box nature of deep learning models for image classification, a popular trend in the literature proposes methods to generate explanations in the form of heat maps indicating the areas that played an important role in the models’ decisions. Such explanations are called saliency maps and constitute an active field of research, given that many fundamental questions are yet to be answered: how to compute them efficiently? How to evaluate them? What exactly can they be used for? Given the increasing rate at which papers are produced and the vast amount of literature that is already existing, we propose our study to help newcomers become part of this community and to contribute to the research field. First, the two existing approaches to generate saliency maps are discussed, namely post-hoc methods and attention models. Post-hoc methods are generic algorithms that can be applied to any model from a given class without requiring fine-tuning. On the contrary, attention models are ad-hoc architectures that generate a saliency map during the inference phase to guide the decision. We show that both approaches can be divided into several subcategories and illustrate each of them with one important model or method. Second, we present the current methodologies used to evaluate saliency maps, including objective and subjective protocols, depending on whether or not they involve users. Among objective methods, we notably detail faithfulness metrics and propose an implementation featuring the faithfulness metrics discussed in this paper ( https://github.com/TristanGomez44/metrics-saliency-maps).

In this era, intelligent vision computing has always been a fascinating field. With the rapid development in computer vision, dynamic gesture-based recognition systems have attracted significant attention. However, automatically recognizing skeleton-based human gestures in the form of sign language is complex and challenging. Most existing methods consider skeleton-based human gesture recognition as a standard video recognition problem, without considering the rich structure information among both joints and gesture frames. Graph convolutional networks (GCNs) are a promising way to leverage structure information to learn structure representations. However, adopting GCNs to tackle such gesture sequences both in spatial and temporal spaces is challenging as graph could be highly nonlinear and complex. To overcome this issue, we propose the spatiotemporal GCNs model to leverage the powerful spatiotemporal correlations to adaptively construct spatiotemporal graphs, called Aegles. Our method could dynamically attend to relatively significant spatiotemporal joints and construct different graphs, including spatial, temporal, and spatiotemporal graph, and well capturing the structure information in gesture sequences. Besides, we introduce the second-order information of the gesture skeleton data, i.e., the length and orientation of bones, to improve the representation of human hands and fingers. In addition, with the public sign language datasets, we use OpenPose technology to extract human gesture skeleton and obtain human skeleton video, building four skeleton-based sign language recognition datasets. Experimental results show that this Aegles outperforms the state-of-the-art ones and that the spatiotemporal correlations effectively boost the performance of human gesture recognition.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

MNA and AS agreed with retraction.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of compromised peer review, which may have led to acceptance of this out-of-scope article. The Editor and publisher no longer have confidence in the results and conclusions of the article.

KY and MJE do not agree with the retraction.

The coronavirus (COVID-19) disease appeared as a respiratory system disorder and has triggered pneumonia outbreaks globally. As this COVID-19 disease drastically spread around the world, computed tomography (CT) has helped to diagnose it rapidly. It is imperative to implement a faultless computer-aided model for detecting COVID-19-affected patients through CT images. Therefore, a detail extraction pyramid network (DEPNet) is proposed to predict COVID-19-affected cases from CT images of the COVID-CT-MD dataset. In this study, the COVID-CT-MD dataset is applied to detect the accuracy of the deep learning technique; the dataset has CT scans of 169 patients; among those, 60 patients are COVID-19 positive patients, and 76 cases are normal. These affected patients were clinically verified with the standard hospital. The deep learning-oriented CT diagnosis model is implemented to detect COVID-19-affected patients. The experiment revealed that the proposed model categorized COVID-19 cases from other respiratory-oriented diseases with 99.45% accuracy. Further, this model selected the exact lesion parts, mainly ground-glass opacity, which helped the doctors to diagnose visually.

Guest Editors Achyut Shankar, Li Zhang, Yu Chen Hu, and Prabhishek Singh introduce the Special Section on Synthetic Aperture Radar Imaging Technology in Deep Learning: New Trends and Viewpoints.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of compromised peer review and determined the paper is unrelated to the special section. The Editor and publisher no longer have confidence in the results and conclusions of the article.

SK disagrees with the retraction. KG, VS, and NS either did not respond or could not be reached.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

HC, QG, AL, and HZ either did not respond directly or could not be reached.

Watermarking consists of embedding in, and later extracting from, a digital cover a design called a watermark to prove the image’s copyright/ownership. In watermarking, the use of deep-learning approaches is extremely beneficial due to their strong learning ability with accurate and superior results. By taking advantage of deep-learning, we designed an autoencoder convolutional neural network (CNN)-based watermarking algorithm to maximize the robustness while ensuring the invisibility of the watermark. A two network model, including embedding and extraction, is introduced to comprehensively analyze the performance of the algorithm. The embedding network architecture is composed of convolutional autoencoders. Initially, CNN is considered to obtain the feature maps from the cover and mark images. Subsequently, the feature maps of the mark and cover are concatenated with the help of the concatenation principle. In the extraction model, block-level transposed convolution and the rectified linear unit algorithm is applied on the extracted features of watermarked and cover images to obtain the hidden mark. Extensive experiments demonstrate that the proposed algorithm has high invisibility and good robustness against several attacks at a low cost. Further, our proposed scheme outperforms other state-of-the-art schemes in terms of robustness with good invisibility.

The real-time detection and recognition ability of human action recognition in a video surveillance system is a key problem in an intelligent surveillance system. Because the behavior recognition for video surveillance systems is affected by the complexity of the scene, the classification performance of the behavior recognition models is not satisfactory. To increase the processing efficiency of the network and solve the problem of low classification accuracy of human action recognition, we designed a deep learning model based on three-dimensional (3D) convolutional network multiscale feature fusion to reduce the impact of constant appearance changes, background clutter, and pedestrian occlusion. The model alternately uses 3D convolution and 3D pooling operations to extract temporal and spatial features between consecutive frames after data preprocessing, and then uses a feature pyramid structure to select three sets of feature layers with different scales. The model performs deconvolution operations in a bottom to up order and fuses with the features of the previous layer, then downsampling and high-level feature layer fusion are performed sequentially from top to bottom. Using the newly generated the highest-level feature layer to realize abnormal behavior recognition. The C3D network algorithm based on feature fusion proposed in this paper is compared with the three most advanced methods of C3D, R3D, and R ( 2 + 1 ) D on the pedestrian abnormal action recognition (PAAR) dataset and the same parameters, and the accuracy is significantly improved.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

DK, SK, SG, SS, VTH, HVV, and AA do not agree with the retraction.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

CH, SH, and YH did not agree with the retraction.

With the development of artificial intelligence and synthetic aperture radar (SAR) imaging technology, methods based on deep learning have been gradually applied to SAR image aircraft detection. However, due to the special SAR imaging mechanism, the imaging results of aircraft targets in SAR images do not have clear features and appear as sparse scattering centers. In addition, airports contain strongly scattering buildings and objects, such as corridor bridges, which will bring huge interference to SAR aircraft target detection. To alleviate these problems, we propose an aircraft scattering information extraction and detection model (SIEDM) based on the spatial imaging characteristics of the targets. First, a scattering center feature extraction module is designed to extract the potential scattering centers of the aircraft targets in the SAR images and acquire the overall structural features of the aircraft targets. In addition, the adaptive noise suppression module is proposed to suppress background interference by learning global information. Finally, we conduct a series of experiments on an SAR image aircraft detection dataset. The experimental results show that the mean average precision of the aircraft SIEDM is 95.7%, which is better than other methods, and the proposed modules in this paper are effective.

Synthetic aperture radar (SAR) image despeckling is a preprocessing method. SAR images are, by default, noisy in nature. The kind of noise found in SAR images is called speckle noise. The effect of this noise on SAR images is highly adverse. It degrades the quality of the SAR image, resulting in the loss of vital information. Since SAR images are inherently speckled in nature, it costs a lot of information loss. The removal of such noise from the SAR image is mandatory and is the first step. The elimination of speckle noise from SAR images is called SAR image despeckling. There are various traditional and nontraditional methods of SAR image despeckling based on Bayesian and non-Bayesian techniques. The SAR image despeckling methods based on Bayesian techniques are further subdivided into spatial and transform domains. This paper presents a comparative review of nontraditional perspectives on SAR image despeckling. The comparison is made based on methodology, objectives, merits, and demerits. Its focus is to do analysis of all the latest research done in the field of SAR image despeckling using nontraditional methods.

Remote sensing technology is constantly developing, which has greatly expanded the use of hyperspectral imaging (HSI). In the arena of remote sensing, the classification of HSI has become a challenging topic. The unique properties of hyperspectral data make accurate categorization difficult. In recent years, deep structured learning has emerged as an effective feature extraction technique for effectively addressing nonlinear problems. It is now extensively used to solve a variety of image processing problems. Deep learning is used to classify images and has shown good performance in recent years. The exact categorization of ground topographies from hyperspectral data is a crucial and current research topic that has gotten a lot of attention. This research work focuses on HSI categorization utilizing several machine learning approaches, such as support vector machine, K-nearest neighbor, and convolutional neural network (CNN). To reduce the number of superfluous and noisy bands in the dataset, principal component analysis and minimum noise fraction (MNF) were utilized. Different performance evaluation measures, such as time taken for testing, classification accuracy, kappa accuracy, precision, recall, specificity, F1_score, and G-mean, were taken to prove the efficacy of the models. The simulation results show that the combination of MNF and CNN produces better classification accuracy compared with the other considered models.

The Editor-in-Chief and the publisher have retracted this article, which was submitted as part of a guest-edited special section. An investigation uncovered evidence of systematic manipulation of the publication process, including compromised peer review. The Editor and publisher no longer have confidence in the results and conclusions of the article.

JY, YZ, LJ, QZ, CY, ZG, and YS either did not respond directly or could not be reached.

Lip reading aims at recognizing texts from a talking face without audio information. Recently, some works have focused on how to effectively extract the spatial information and temporal information. We introduce an innovative two-stream network to make full use of the complementarity of global spatial information and local spatial information. The global spatial information is directly generated by the global stream. Furthermore, we design a patches selection module in the local stream to conveniently select the critical local information using attention mechanism. Then, the fusion features of the two streams and the global features are fed into the temporal module to further explore the temporal clues. To guide the selection of the local information from the fused features and to make the global stream and local stream learn from each other, we design a global information guide loss and a mutual learning loss, respectively. Finally, extensive experiments on both LRW and CAS-VSR-W1K datasets demonstrate the superiority of our two-stream work.

With the widespread of digital information and advanced internet technologies, cyber-physical systems (CPS) generate numerous data. Due to the limited storage and computing capacity, CPS devices outsource data to cloud servers. When the data is outsourced to cloud servers, users lose control of their data. Several traditional and blockchain auditing schemes were proposed to verify the integrity of the data. Unfortunately, these existing schemes suffer from key escrow problem, and they are unsuitable for CPS devices as they involve substantial computation costs. This paper proposes an efficient blockchain-based certificateless short signature public auditing scheme for cloud-based CPSs, which uses blockchain technology to verify the integrity of the data. Further, our scheme requires only one pairing operation and two hash functions in the verification process. The performance analysis demonstrates that the proposed scheme outperforms and satisfies low-bandwidth device requirements.

Nowadays, the rapid growth of personal handheld electronic devices encourages the individual and organizations to upload their information to cloud for storage and processing purposes. To ensure privacy, the images are encrypted using cryptographic schemes that are outsourced to the cloud. Though images are encrypted, searching for similar images leads the cloud server to access the image data as it performs computation over plaintext. Thus, to ensure the privacy of the images during image retrieval, the proposed framework presents a transfer learning-based secure quantum image retrieval scheme over encrypted cloud. The confidentiality of the images is guaranteed by introducing quantum-based image encryption. Meanwhile, clustered image feature vectors are extracted through the transfer learning model and protected using secure multiparty computation. During retrieval, the proposed system introduces a similarity comparison model for performing computation on encrypted data. The experiments and performance analysis show the effectiveness and security of the proposed scheme.

Simple demosaicking methods designed for linear CMOS image sensors, such as MATLAB’s demosaic function, may be used with a monotonic nonlinear sensor having a Bayer color filter array (CFA). However, such methods may be inadequate at handling dynamic salt-and-pepper noise (SPN), i.e., outlier pixels, which is expected in images taken with a nonlinear sensor. Although SPN is present with linear sensors, nonlinear sensors express low-density light-dependent SPN that requires filtering. Extending a recent work on dynamic SPN filtering of a nonlinear sensor, we propose, evaluate, and verify a nonlinear method and apparatus to demosaic images, taken with a Bayer CFA, while simultaneously filtering the SPN. The approach relies on the use of weighted medians to filter the SPN, especially at densities that imply isolated outliers in small neighborhoods, while determining an accurate red, green, and blue (RGB) color at every pixel location. For explanatory purposes, three variants of the proposed method are presented and evaluated. A ground-truth image set, in which RGB channels were not obtained by demosaicking, is subsampled in a Bayer CFA pattern to produce mosaicked images for testing. In varying densities, SPN is introduced to these Kodak images for method and apparatus evaluation. Results of the proposed method and its variants are compared with those obtained with MATLAB’s demosaic function. Considering the alternatives and also apparatus complexity, the proposed nonlinear demosaicking method proves superior in visual quality, for smooth textures and along edges, and in peak signal-to-noise ratio, when there is low-density SPN.

Although existing methods have achieved impressive accomplishments in the field of low-light image enhancement, the phenomenon of overenhancement remains a challenge. To address this issue, a multiport output enhancement structure combined with multiloss functions supervision is designed to obtain multiple images with different enhancement intensity. Subsequently, an enhancement estimation module is proposed to flexibly select the most suitable enhancement image from these series of enhanced images, thereby reducing overexposure. To this end, the enhancement estimation network (EENet) for enhancing low-light images is introduced. The proposed EENet enhances low-illumination images with different brightness levels more perfectly because the phenomenon of overenhancement is reduced. For qualitative comparison, the experimental results demonstrate that the proposed method enhances low-illumination images more naturally, especially for enhancing low-illumination images that have brighter illumination. For quantitative evaluation, the proposed method obtains the highest peak signal-to-noise ratio and structural similarity on the public single image contrast enhancer dataset and reconstructed low-light* dataset compared with those listed methods. In addition, the EENet was proven to outperform state-of-the-art methods in dark image face detection, indicating that the EENet has great practical potential.

As a major research field in computer vision, automatic generation of Labanotation has been attracting the attention of many researchers. These researchers use various methods to engage in dance education and protection, but the existing methods do not consider the spatial-temporal dependence modeling of dance movements. Therefore, they do not represent complex dance movements optimally. We propose an automatic generation network of Labanotation based on the long and short spatial-temporal relations. It includes local spatial-temporal feature extraction network, global spatial-temporal feature extraction network, and local and global feature fusion network. It can align the input and output sequences while modeling the skeletal spatial-temporal relation. The local spatial-temporal features among short interval skeleton sequence relationship are obtained through the multi-scale convolution of time and space. The global spatial-temporal features are learned through the transformer network to obtain the relationship among skeleton sequences with long interval. For the output of the two networks, we use the pyramid squeeze attention network to exchange information in long and short spatial-temporal information, achieving complementarity to improve the accuracy of action recognition. Experimental results show that the proposed method outperforms the state-of-the-art methods on laban16 and laban48, which are common datasets for the research of Labanotation.

When an image translation task contains intradomain translations, the untranslated source image will be discriminated as the real by the discriminator. Thus if the network’s nonlinearity is insufficient, the generator can fool the discriminator by producing output that resembles the source image. We propose an activation function termed “adaptive rectified linear unit (ReLU) with structure adaption (SA-AdaReLU)” to enhance the control and nonlinearity of the network in image translation tasks. SA-AdaReLU is composed of two technologies: adaptive ReLU (AdaReLU) and structural adaptive function. The proposed AdaReLU can dynamically change the channel-wise data distribution to better utilize the features in negative regions, which helps to improve the control of the network when inner-domain translation is involved. Meanwhile, the structural adaptive function further enhances the feature selection ability of adaptive instance normalization (AdaIN) and enhances the network’s spatial nonlinearity to manipulate the spatial structure on the feature maps. Extensive experiments have demonstrated the effectiveness of the proposed SA-AdaReLU. In addition, with SA-AdaReLU, fewer layers are required to achieve the same visual effect for building the generator, thus reducing the computational complexity.

The advent of deep learning has enabled image super-resolution (SR) technology to achieve tremendous success. The recent exploration between convolutional neural network (CNN) and transformer has produced an impressive performance. However, the huge model size and heavy computational cost occupied by most approaches cannot be ignored. Additionally, the global and local modeling of image features is not well utilized and explored, which is important for texture content reconstruction. In this study, we present an efficient and lightweight network based on CNN and transformer for image SR, dubbed dual-aware transformer network (DATN). Specifically, a dual-aware fusion module (DAFM), consisting of spatial-aware adaptive block (SAAB) and global-aware fusion block (GAFB), learns local and global features in a complementary manner for yielding abundant content details. SAAB delivers the captured spatial information to GAFB in parallel, whereas GAFB establishes long-range spatial dependencies among diverse features to boost the network’s ability for recovering texture details. Further, in the upsampling stage, our proposed transformer-empowered upsampling module aggregates global information from the transformer module to generate location-wise reassembly kernels for content-aware upsampling. In such way, feature-rich information is highlighted adaptively to reconstruct high-quality images with maximum efficiency. Experimental results reveal that our proposed DATN achieves high qualitative performance while maintaining low computational demands, surpassing most state-of-the-art SR networks.

In the field of three-dimensional (3D) reconstruction, learning-based neural radiation field (NeRF) and 3D-aware generative model (3D-aware GAN) set the foundation for implicit reconstruction of a 3D object. However, this method has problems of slow reconstruction speed and uncontrollable reconstruction results. For this reason, we proposed Info-Giraffe, a virtual neural feature field generation method for object in limited views condition. Info-Giraffe is divided into two phases: pretraining and retraining. First, we generate the basic neural feature field of the object using the simplified Giraffe model in the pretraining phase. Then, during the retraining phase, we extract the 3D information of the object in limited views using a set of latent code encoders designed based on InfoGAN, which is then reparameterized into the latent space. Finally, we continue migration learning by introducing 3D information on the foundation of the object’s basic neural feature field and generate a controllable virtual neural feature field of the object in limited views, which enables fast 3D reconstruction of the object and control of the reconstruction results. Experimental results on Chairs 64² and CompCars 256² natural image datasets reveal the effectiveness of the proposed method.

In the process of image stitching, many problems will inevitably arise, such as misalignment, artifacts and local structure distortion in the overlapping regions. A parallax image stitching algorithm combining improved feature optimization with an innovative iterative seam estimation is proposed. First, the point features and line features of input images are detected. To optimize point features, the histogram statistical approach is proposed to remove false matching points combined with RANSAC algorithm. Second, the mesh warp is optimized by minimizing the sparse quadratic total energy function so as to achieve accurate alignment in the overlapping regions. Finally, we propose an iterative seam estimation method using an improved quality evaluation strategy. Experimental results show that our method has higher matching accuracy and visually better image stitching performance than other methods.

Fine-tuning a pretrained model with real data for a machine learning task requires many hours of manual work, especially for computer vision tasks, where collection and annotation of data can be very time-consuming. We present a framework and methodology for synthetic data collection that is not only efficient in terms of time taken to collect and annotate data, making use of free- and open-source software tools and 3D assets but also beats the state-of-the-art against real data, which is the ultimate test for any similar-to-real approach. We test our approach on a set of image classes from ObjectNet, which is a challenging image classification benchmark test dataset that is designed to be similar in many respects to ImageNet but with a wider variety of viewpoints, rotations, and backgrounds, which can make it more difficult for transfer learning problems. The novelty of our approach stems from the way we create complex backgrounds for 3D models using 2D images laid out as decals in a 3D game engine, where synthetic images are captured programmatically with a large number of systematic variations. We demonstrate that our approach is highly effective, resulting in a deep learning model with a top-1 accuracy of 72% on the ObjectNet data, which is a new state-of-the-art result. In addition, we present an efficient strategy for learning rate tuning that is an order of magnitude faster than regular grid search.

Video super-resolution (VSR) has been improved with various deep learning architectures and datasets that mostly contain clean images. However, most real-world videos are compressed and here arises a critical issue: each frame in a video usually varies in quality. We discover an important but simple coding prior that affects the performance of the existing VSR models because the prior can tell us which frame is outstanding than others in terms of quality, namely outstanding-frames. Exploiting the prior, we propose a method that allows you only look outstanding frames (YOLOF) to enhance the existing VSR models as a universal approach, which feeds VSR models the best quality of frames near the reference frame with given distance. Extensive evaluations with various VSR models show that our YOLOF method enhances existing VSR models substantially without harming original architectures.

In practical applications, it is urgent to develop lightweight, fast, and accurate face detectors. Although the performance of face detection has made tremendous progress with the rapid development of deep learning, they still struggle to meet the requirements of both efficiency and effectiveness. This is because current face detectors either use large networks or cumbersome processes oriented for accuracy but leading to heavy model complexity and slow inference speed or use various simple designs oriented for efficiency but greatly sacrificing accuracy or not actually efficient. In this paper, we conduct an in-depth study of the network structure and anchor setting and present an efficient and effective face detector named EEFDet, which can detect faces with lightweight model, fast inference speed, and excellent accuracy. First, a lightweight, fast, and robust network (LFRNet) is constructed by carefully designing the backbone network, detection neck, and detection head of the network. LFRNet not only has small model complexity and low latency but also possesses robust feature extraction ability. Then, a sparse anchor strategy is proposed by rationally setting anchors associated with the detection head to avoid using too many redundant anchors and get the most effective anchors. This strategy enables the detector to process multiple anchor-related parts at a fast speed while still guaranteeing detection accuracy. Extensive experiments and detailed ablation analysis are conducted on several widely used face detection benchmarks based on multiple metrics and different platforms. The results show that our EEFDet is lightweight (1.195 M parameters and 0.719 G FLOPs), fast (38.780 ms on GPU and 141.787 ms on CPU), and accurate [90.8%/89.6% (easy), 88.1%/87.2% (medium), 77.9%/77.4% (hard) APs on Val/Test set of WIDER FACE, 96.3% discrete score and 79.7% continuous score on face detection data set and benchmark (FDDB), 97.7% AP on PASCAL FACE, and 98.2% AP on annotated face in-the-wild (AFW)], achieving an optimal trade-off between efficiency and effectiveness compared to state-of-the-art face detectors.

Image dehazing has been widely used in vision-based fields, such as detection, segmentation, traffic monitoring, and automated vehicle system. However, most of the existing end-to-end dehazing networks are fully data driven without physical constraints or prior information guidance, leading to difficulties in exploring latent structures and statistical characteristics of hazy images. We propose a novel Retinex decomposition-fusion dehazing network consisting of a dual-branch decomposition module and a fusion optimization module. Different from existing solutions, we decompose the clear images in the commonly used RESIDE dataset based on Retinex theory to construct the clear illumination map and reflection map datasets to drive the network training, equivalently imposing reasonable constraints on the network and achieving impressive dehazing performances. The dual-branch decomposition module is developed to estimate the illumination map and the reflection map, respectively. The illumination map mainly contains the global features of the image, whereas the reflection map reflects the inherent color properties of the image and contains rich details, with which we explore the latent structures and statistical characteristics of hazy images. In addition, the dual-branch structure avoids the error accumulation and information cancellation existing in current methods. Subsequently, the estimated illumination map and reflection map are fused and refined via the fusion optimization module to access the dehazed image. Experiments show that the proposed network has better generalization and visual effects than existing fully data-driven methods and can be applied successfully to real-world scenarios.

Most current approaches in the literature of scene text recognition train the language model via a text dataset far sparser than in natural language processing, resulting in inadequate training. Therefore, we propose a simple transformer encoder–decoder model called the multilingual semantic fusion network (MSFN) that can leverage prior linguistic knowledge to learn robust language features. First, we label the text dataset with forward, backward sequences, and subwords, which are extracted by tokenization with linguistic information. Then we introduce a multilingual model to the decoder corresponding to three different channels of the labeled dataset. The final output is fused by different channels to get more accurate results. In experiments, MSFN achieves cutting-edge performance across six benchmark datasets, and extensive ablative studies have proven the effectiveness of the proposed method. Code is available at https://github.com/lclee0577/MLViT.

Conventional deep neural networks (DNNs) have been shown to be vulnerable to images with adversarial perturbations, referred to as adversarial examples. In this study, we propose a method to protect neural networks against adversarial examples using perceptual image hashing. Because perceptual hashing is robust to adversarial perturbations, we combine hash sequences of input images with the parameters of a neural network in an image-hash processing network. Thus, outputs of the neural network are affected by image hashes, which render the model robust to adversarial examples to some extent. Thus, the proposed approach provides a defense against adversarial examples. The experiment was conducted on the CIFAR-10 dataset, and we used ResNet-18 as our target network. To verify our method, we tested our defense network using several common white-box attacks and black-box attacks. The results show that it achieved a significant improvement in the classification accuracy for adversarial examples.

Video super-resolution (VSR) is a process of high-resolution reconstruction of low-resolution video. To address the problems of the previous VSR methods with poor temporal consistency and unsatisfactory SR results, we proposed a nonlocal deformable alignment and frame recursive progressive fusion (RPF) network combining sliding window and recursive methods, which uses nonlocal operations to align sequential frame features and later applies recursion to temporally model the hidden information and alignment features of the previous moment, thus improving temporal consistency. The RPF unit is used to fully fuse the hidden information with the currently aligned features, acquiring more supporting information to be obtained from adjacent frames, resulting in better SR results. The results were evaluated on the three public VSR datasets of Vid4, udm10, and Vimeo-90K, and the experimental results show that the proposed method can achieve state-of-the-art performance on VSR task.

Substantial research has explored methods to optimize convolutional neural networks (CNNs) for tasks such as image classification and object detection, but research into the image quality drivers of computer vision performance has been limited. Additionally, there are indications that image degradations such as blur and noise affect human visual interpretation and machine interpretation differently. The general image quality equation (GIQE) predicts overhead image quality for human analysis using the National Image Interpretability Rating Scale, but no such model exists to predict image quality for interpretation by CNNs. Here, we assess the relationship between image quality variables and CNN performance. Specifically, we examine the impacts of resolution, blur, and noise on CNN performance for models trained with in-distribution and out-of-distribution distortions. Using two datasets, we observe that while generalization remains a significant challenge for CNNs faced with out-of-distribution image distortions, CNN performance against low visual quality images remains strong with appropriate training, indicating the potential to expand the design trade space for sensors providing data to computer vision systems. Additionally, we find that CNN performance predictions using the functional form of the GIQE can predict CNN performance as a function of image degradation, but we observe that the legacy form of the GIQE (from GIQE versions 3 and 4) does a better job of modeling the impact of blur/relative edge response in our experiments.

To resolve the problem of occlusion of the depth information of x-ray images and the detection of small-scale contraband in the detection of contraband objects, an improved prohibited item detection network has been proposed based on YOLOX. First, a material-aware atrous convolution module (MACM) is added to the feature pyramid network to enhance the model’s multiscale fusion and extraction ability for material information in x-ray image. Second, a spatial pyramid split attention mechanism (SPSA) is proposed to fuse spatial and channel attention for different scale spatial information features. Finally, CutMix data augmentation strategy is adopted to improve the robustness of the model. The overall performance identification experiments were conducted on the publicly available OPIXray dataset. The average accuracy (mean average precision, mAP) of the method is 93.10%. Compared with the baseline model YOLOX, the mAP is improved by 3.25%. The experimental results show that our method achieves state-of-the-art detection accuracy compared with existing methods.

Simultaneous localization and mapping (SLAM) technology has gained popularity as a result of technological advancements, and image matching—the labor-intensive and crucial component of SLAM technology—has emerged as a key area of research. Hence, the purpose of this study is to develop an image matching method based on improved accelerated-KAZE (AKAZE) and Gaussian mixed model (GMM) segmentation to guarantee the successful completion of subsequent vision challenges. (i) For feature extraction and feature description, first, the AKAZE algorithm was used to extract the key points of the gray-scale image, and second, the binary robust independent elementary features (BRIEF) algorithm was used to obtain the descriptors. Introduction of gray-scale prime method and convolution kernel function to enhance the robustness of features in the BRIEF calculation process. (ii) For feature matching, the gray-scale image is pseudocolored, and then all the pixels in its R channel are segmented by GMM, retaining the features in the same segmented region after brute-force matching. The experimental results demonstrate that, in comparison to oriented FAST and rotated BRIEF and AKAZE, the proposed algorithm in this work provides higher matching accuracy for images of different data sets. That is, the matching approach can be used more effectively for subsequent vision tasks.

Low-rank and sparse decomposition (LRSD) plays a vital role in foreground–background separation. The existing LRSD methods have the drawback: imprecise surrogate functions of rank and sparsity. We propose the weighted Schatten p-norm (WSN) and Laplacian scale mixture (LSM) method based on LRSD for foreground–background separation, which introduces the WSN and LSM to improve this drawback. To demonstrate the performance of the proposed method, it is applied to foreground–background separation and gets the highest average F-measure score.

Jointly learned detectors and descriptors are becoming increasingly popular because they can simplify the matching process and obtain more correspondences than traditional tools. However, most methods yield low keypoint detection accuracy due to the large receptive field of the detection score map. In addition, existing methods lack efficient detector loss functions because the coordinates of keypoints are discrete and nonderivable. To mitigate these two problems, we propose a method called dynamic attention-based detector and descriptor with effective and derivable loss (DA-Net). For the first problem, a dynamic attention convolution-based feature extraction module is proposed to select the most suitable parameters for different samples. In addition, a multilayer feature self-difference detection (MFSD) module is proposed to detect keypoints with high accuracy. In the MFSD module, multilayer feature maps are used to calculate their feature self-difference maps, and they are fused to obtain a detection score map. For the second problem, an approximate keypoint distance loss function is proposed by approximately regressing the coordinates of the local maximum as keypoint coordinates, allowing the calculations involving keypoint coordinates to backpropagate. Moreover, two descriptor loss functions are proposed to learn reliable descriptors. A series of experiments based on widely used datasets show that DA-Net outperforms other learned detection and description methods.

Adversarial example generation (AEG) has been a hot spot in recent years because it can cause deep neural networks (DNNs) to misclassify the generated adversarial examples, which reveals the vulnerability of DNNs, motivating us to find good solutions to improve the robustness of DNN models. Due to the extensiveness and high liquidity of natural language over the social networks, various natural language-based adversarial attack algorithms have been proposed in the literature. These algorithms generate adversarial text examples with high semantic quality. However, the generated adversarial text examples and the corresponding attack models may be maliciously or illegally used. To tackle this problem, we present a general framework encapsulated in the cloud application programming interfaces (APIs) for generating watermarked adversarial text examples to protect adversarial text examples and corresponding adversarial text attack models. For each word in a given text, a set of candidate words are determined to ensure that all the words in the set can be used to carry secret bits or facilitate the construction of adversarial example. By applying a word-level adversarial text generation algorithm, the watermarked adversarial text example can be finally generated. Experiment results show that the adversarial text examples generated by the proposed method not only successfully fool advanced DNN models, but also carry watermarks that can effectively verify the ownership and trace the source of the adversarial examples and the corresponding attack models. Moreover, the watermark can still survive after attacked with AEG algorithms, which has shown the applicability and superiority.

Remote photoplethysmography (rPPG) is a video-based heart rate measurement technology, which is widely used in special scenes where contact equipment is difficult to measure. However, the rPPG signal is very weak, and it is easily affected by factors such as uneven environmental illumination changes and the tester’s head movement, which leads to the poor robustness of the existing methods in natural scenes. A deep learning model based on vision transformer is proposed to segment the facial skin region to generate the spatiotemporal feature map of the video sequence, inputs the feature map into the model for rPPG physiological feature extraction, and then fits the rPPG signal. The experiments verify the effectiveness of the method on mixed data sets and can ensure that the model has a high degree of signal fitting while significantly reducing the computational complexity of the transformer.

Existing methods for object detection in road marking images ignored an important challenge—imbalanced class distribution in road marking images—which lead to poor performance on tail classes. Existing approaches to this issue focus mostly on data quantity. However, throughout the training process, the quantity and difficulty of each class are two related and equally important problems. To this end, we propose a framework, tripling sampler, and head detection network (TSHNet), which consists of class-preference samplers (CPS) and trilateral box heads (TBH). The CPS is composed of two complementary factors: the quantity factor and the difficulty factor. TBH is designed to handle tail&hard classes, common classes, and head&easy classes in a triple-path manner. We evaluate our approach on CeyMo and road marking datasets and achieve excellent performance when combined with PolyLoss. Our results demonstrate that TSHNet significantly outperforms base detectors and generic approaches for long-tail road marking problems.

The human brain needs to rely on the information obtained by the eyes to process information. To a certain extent, the eyes are the reflection of the human brain activity. Based on this, many people pursue eyeball changes to judge a person’s psychological state, and eye tracking technology has also been proposed, playing an increasingly important role in different fields. The purpose is to study the human attention distribution system in the computer environment. We use the eye tracking technology of the eye tracker to record any actions of each learner during learning, use the function of the eye tracker to obtain various data, and use statistical analysis to analyze it. The experimental results show that the eyes can reflect human behavior information well. Analyzing eye movement data can deepen people’s understanding of the mechanism of attention distribution and improve learning efficiency by 20%.

Underwater bionic robots are very similar to real underwater creatures in appearance and features. This makes it difficult for autonomous underwater vehicles (AUVs) to distinguish their appearance features. Moreover, the existing underwater target detection methods have large model sizes and many parameters. This leads to difficulty in practical application of such UAV methods. To solve this, we propose an underwater camouflaged target recognition algorithm based on an adversarial graph structure. First, the salient features and environmental features of underwater images are fully explored by a custom salient feature extraction network and a related environmental feature extraction network. Second, the camouflaged target graph structure and the natural target graph structure of the fusion environment feature are constructed to obtain the dependency between the target and environment features. Finally, an adaptive frame regression camouflaged target detection network is constructed based on adversarial learning to recognize camouflaged targets and natural targets. In contrast to the existing target recognition model in the simulation of the four datasets of concealed target detection, quantitative underwater image dataset, underwater image enhancement benchmark dataset, and underwater image dataset, the proposed model gets higher average recognition accuracy of 0.820, which is an improvement of 3.4%, and the number of parameters is only 1.2 m. Proved by the experiments, the proposed target recognition algorithm is more effective and superior to the existing algorithms.

Multi-focus image fusion is the process of fusing images of the same scene with different focus ranges. However, pixel misclassification and artifacts caused by noise have been unavoidable problems in fusion methods. To overcome this problem, we propose gradient tensor high-order singular value decomposition (HOSVD) to fuse multi-focus images. The eight-direction Sobel gradient operator is used to obtain second-order gradients in different directions. Tensor-based information processing can effectively represent high-dimensional information and extract structural information from multi-focus images. Therefore, second-order gradient matrices of different images are formed into tensors. The gradient tensor of the images is transformed by HOSVD to obtain the core tensor and factor matrixes. The images are better represented by multiplying the core tensor with the factor matrix corresponding to the direction of the image. In the core tensor, the larger singular values represent basic information and the small singular values high-frequency information. The noise generally belongs to high-frequency information, so we select the larger singular values in the core tensor as activity level variables of the image to reduce noise. Experimental results demonstrate that the proposed fusion method can effectively reduce noise effects and achieve state-of-the-art fusion performance in both objective metrics and visual quality.

As power grid structure becomes increasingly complex, the impact of the operation status of key substation equipment on the safe and stable operation of the power grid is gradually increasing. Aiming at the problems of dense targets and occlusion recognition difficulties in the complex background of substations, an improved you-only-look-once (YOLO) v4 insulator defect detection algorithm is proposed. A large number of insulator and defect datasets were collected from the Internet. We use MobileNet-v2 to replace the backbone network of YOLOv4, and in view of the problem that the accuracy of YOLOv4 target detection model is not high for small targets or dense targets, convolutional block attention module (CBAM) attention mechanism is added to the backbone network to enhance the feature extraction ability, and ultra-lightweight subspace attention module (ULSAM) is added to the detection head to learn the cross-channel information. Experiments based on InDataset show that the insulator defect detection algorithm based on deep learning proposed achieves a high detection accuracy and detection speed; the mean average precision of the improved YOLOv4 model is 87.48%, and the recall rate of insulator defects reaches 79.84%, which meets the requirements of robustness and accuracy of insulator defect detection in real scenarios.

Unsupervised person reidentification (re-ID) is designed to deal with the problem that in industrial application scenarios, the consistent features of the same person cannot be fully mined due to the lack of annotated information in the collected image resources. Researchers focus on using pseudolabels to label images with the same clustering attributes, but traditional clustering methods are prone to generating noisy pseudolabels, which greatly reduce the accuracy of unsupervised person re-ID. We propose a method to refine pseudolabels by generating discriminative information based on channel partitioning. In the design process, the feature map is divided from the channel level, generating a proximity matrix by calculating the distance between the channel feature and the global feature, and select the most negative sample as the verification label to optimize the pseudolabel of the global feature. As a bidirectional guide, the global pseudolabel can be used to further smooth each channel label, keeping the consistency of channel features and global features in pseudolabel selection. This unique method of learning local features can link global information and channel information. Experimental results demonstrate the superior performance of our proposal on representative person re-ID datasets.

Recent research on video violence detection has made great progress with the development of multi-modal fusion techniques. However, existing approaches still pose huge challenges in real-time violence detection due to fusion features being fixed and not able to be further fine tuned. We aim to address this challenge by exploring the multi-modal fusion method for analyzing multiple modal violent information, i.e., audio, optical flows, and RGB images. We propose a unified network called VioNets, which contains both a cross-attention graph convolutional network (GCN) module and a bidirectional gate recurrent unit (Bi-GRU) module for fusing different modalities of information. First, the cross-attention GCN module is utilized to extract the cross-modal spatial–temporal features. The Bi-GRU module is then applied to accurately capture both past and future context features for each time step of the single-modal features. As a result, the model retains important single-modal information in the extracted features while using the cross-modal features to improve the detection accuracy. Experiments conducted on the XD-Violence dataset show that the proposed method achieves an average precision of 80.59% and an inference time of 0.16 s with 1.82 M parameters.

The single shot multibox detector (SSD) is one of the most important algorithms in single-stage target detection, having the characteristics of multiscale detection and rapid detection speed. However, the effective SSD feature layers are independent of one another, which can lead to object detection difficulties. To address this problem, we proposed an improved SSD object detection algorithm. First, the global attention mechanism (GAM)—which can enhance spatial and channel information—was introduced into the multiscale feature layer. The channel attention module of the GAM was improved. Second, a feature fusion module was introduced to strengthen the relationship between feature layers. Finally, the cross stage partial structure was introduced into the feature fusion module, and used to improve the model’s learning ability. For model training and detection based on the PASCAL VOC dataset, the mean average precision and frames per second obtained by the improved SSD algorithm were 84.67% and 18.67, respectively, which could effectively detect difficult targets.

In recent times, deep neural networks (DNN) have been extensively used in several areas and have achieved great success. State-of-the-art face recognition (FR) systems have gained high accuracy using DNN. However, researchers have found that DNN-based systems fail when facing an adversarial attack on images. In adversarial attacks, the adversary modifies the face images in a manner that the human does not perceive the changes in the generated image but FR systems are unable to recognize the faces correctly. We proposed a method to generate an adversarial attack. A multistack adversarial network (M-SAN) patch-based attack is generated using the generative adversarial network under the black-box settings. The M-SAN attack targets the features of the face images using a patch to fool the FR model as targeted and untargeted attacks. In the past, several attack generation methods have been presented under white-box settings. However, white-box settings need a model architecture and information about its parameters. Due to this, a single white-box attack is not able to fool different FR models. We propose an attack generation approach that is based on black-box settings in which an attacker does not have access to the target model parameters. The attack is generated with the help of the surrogate model and then transferred to the various target models. The proposed M-SAN attack is applied to FR models including FaceNet, ArcFace, and CosFace on the labeled face in the wild dataset.

A progressive histogram equalization (HE) and contrast enhancement method is proposed in the frequency domain. This method has an iterative structure based on the 6-sigma rule in the low-dynamic range using discrete wavelet transform. The proposed method determines an automatic threshold for attenuating the high-frequency amplitude of a signal in the discrete wavelet domain, based on the standard deviation of the absolute power of the high-frequency components in the signal band generated from the probability density function (PDF). Then an iterative quantization procedure is used where the PDF values in the frequency domain are randomly quantized. Besides, the maximum Bhattacharyya coefficient, which matches the original input image’s PDF, is used as a cost function to optimize the histogram smoothing output. For the random number generation, a 32-bit pseudorandom generator is employed to produce the same result for the output image at each runtime. The quantization factor parameter enables a controlled contrast enhancement rate to receive balanced equalization results in images. In the experimental studies, the proposed iterative HE method is compared with the well-known global, local, and brightness preserving algorithms. Experimental studies quantitatively and qualitatively display promising and encouraging results in terms of various state-of-the-art quality assessment metrics such as mean squared error, peak signal-to-noise ratio, structural similarity index measurement, contrast improvement index, absolute mean brightness error, and quality-aware relative contrast measure.

Image aesthetic assessment (IAA) is a challenging task in computer vision fields, which aims to automatically evaluate image beauty by simulating human perception on image aesthetic. With the development of deep learning, although convolutional neural network (CNN)-based IAA approaches have achieved extraordinary progress, CNN experiences difficulty to capture long-distance relationships among visual elements. There is a strong correlation between image layout and image semantic information for image aesthetic. In order to solve this problem, an another scale-guided parallel transformer is proposed, including a multiscale local feature extractor (ME), a feature projection (FP), and an another scale-guided parallel feature fusion transformer (AST). The ME captures primary local features with classic ResNet at multiple scales. The FP performs dimension transformation on feature maps for each scale, which can obtain feature token and aesthetic token. The AST with two parallel transformer encoders is exploited to highlight the significant regions in the holistic image, in which the feature tokens and the aesthetic token from another scale are grouped together to obtain interscale guidance. The final score distribution is achieved by weighting multiple aesthetic tokens with learnable parameters for unified aesthetics assessment. Extensive experiments on two public datasets, including aesthetic visual analysis and aesthetics and attributes database, demonstrate that the proposed method outperforms the state-of-the-art methods across three different tasks.

In the present era of big data applications, incremental learning has emerged as the most admired area of research where ever-ending tasks from different application domains arrive temporarily, and the learning models majorly focus on newly arriving tasks while forgetting the historically acquired knowledge of old tasks. In this context, when a deep neural network (DNN) performs a sequence of learning tasks, it usually forgets the previously learned tasks. This phenomenon is known as catastrophic forgetting (CF), and the existing state-of-the-art deep learning approaches are unable to handle this issue. Following this, many DNN-based state-of-the-art solutions have been proposed to alleviate the problem that ensures a more natural way of human brain imitation. During training, deep learning algorithms are subjected to a random sequence of diversified datasets (also referred to as cross-domain datasets) that typically belong to multiple domains and have a variety of feature representation, distribution, density, etc. In such cases, CF may become a serious issue because the model tends to forget the previously learned features that affect the overall performance. Because there has been very little research in this area, a more robust incremental deep learning algorithm is proposed in our study to learn desired tasks incrementally. To overcome this situation, proper sequencing of cross-domain datasets is important so that related datasets are kept together to enhance the learnability of the model. Our research proposes a cross-domain order approach with hard attention to the task (HAT) methodology that uses an attention-gated mechanism to efficiently learn the task-specific features. Intense experiments have been performed, and the performance of the proposed approach is compared with the changing cross-domain orders with the baseline models over the following datasets: MNIST, P-MNIST, CIFAR-10, CIFAR-100, and SVHN, etc. The results of the experiments show that learning through cross-domain orders results in better incremental learning.

Object detection is an enabling technology of computer vision for locating instances and categorizing classes of objects in images or videos. It has made enormous strides with significant growing of deep learning research over the last decade. However, there are severe challenges on small object detection in complex scenes as they appear with low resolution and have not enough contrast from the background information. This disturbance may cause missed detection and detection accuracy decline for small objects. We propose a small object detection network in the dark light scene based on improved YOLOv5. The network takes YOLOv5 as the baseline and incorporates a channel and spatial dual-branch backbone module, which enhances the details by fusing the features of the two branches. We also introduce a densely linked feature fusion network before detection layers with receptive field block. The fusion network integrates deep features with shallow ones across different scales. A data augmentation module is used to enhance the brightness and limit the contrast for small object detection in the dark light scene. Experiments are carried out on the Dark Face and darkened Vis Drone dataset. The results show that the evaluation index of the proposed method is better than that of the comparison methods. From the detected images, it is obvious that the undetected frame of the small object in the dark light scene decreases and the detection accuracy improves. All of the results show that our model has better performance than some existing methods for small object detection in the dark light scene.

Different from generic image classification, fine-grained classification, such as facial expression classification, in which multiple expressions share inherently similar underlying facial appearances, may show a small difference between facial expression classes. Unlike lab-controlled data, facial expressions from natural scenes have rich forms of the same expression due to the diversity of subjects and the complexity of real-world conditions, and as a result, facial expressions may have large differences among samples within the same class. Moreover, there is little difference between facial expressions, and facial expressions are displayed simultaneously through various facial regions, which require us to encode the feature of multiple key regions, forming high-order interactive information. To address the aforementioned problems, we design an enhanced capsule network based on multi-level feature fusion attention mechanism, which is comprised of four critical components: multi-level feature extraction module (MFEM), multi-level attention module (MAM), multi-level capsule attention fusion module (MCAFM), and reconstruction module (RM). The MFEM collects the low-level, middle-level, and high-level features from the input image, therefore lowering the high-level convolution layer’s susceptibility to blurred image and the problem of pose variation. The MAM directs the network’s attention to the most significant features in different levels of image features and can assist the network in ignoring blurred, occluded, and irrelevant features and incorporating them into our self-attention center loss function to compress the element distribution in the same class. The MCAFM preserves the attributes of each face region (such as location, size, and direction) by transferring the features into capsules in preparation for the eventual creation of the dynamic routing mechanism, which can resolve the problem of image rotation on FER in the wild. Simultaneously, the capsule features of distinct areas are combined to provide higher-order overall feature information, enhancing the model’s capacity to discriminate between different kinds of expressions. The RM reconstructs the image and calculates the difference between the reconstructed image and the original input image. Our model outperforms a large number of current methods on two public datasets, RAF-DB and SFEW.

The railway system of Japan plays a vital role in the national transportation network. A key issue in public transport safety is anomaly detection in railways. Lately, developing robust algorithms and methods for anomaly detection has become the premier task in this field. Recently introduced approaches based on convolutional neural networks, generative adversarial networks, and vision transformers (ViTs) have remarkably improved the research in anomaly detection. Our work proposes a high-performance module for the anomaly detection of signal bonds. First, we present an overview of the proposed module; then, the object detection model and the proposed hybrid classification model based on texture-biased convolutional neural networks and edge-biased ViTs are introduced. Finally, the proposed anomaly detection module is evaluated for accuracy using the dataset from the East Japan Railway Company and the receiver operating characteristic curve. The results show that our proposed module achieves a better performance on real-life data than the two methods that were combined to generate it, raising hope toward possible usage in other areas as well.

Accurate segmentation of road cracks is vital for identifying cracks in pavements for greater traffic safety. However, owing to blurred boundaries, low contrast between cracks and the surrounding environments, changes in color and shape, most existing segmentation methods face significant challenges in obtaining receptive fields and extracting image feature information. To overcome these issues, we constructed a new framework, named ASARU-Net, to analyze and segment road cracks. We utilized a superimposed U-Net architecture instead of the original 3 × 3 convolution layer to improve the receptive field and enhance segmentation performance. Then, we employed a convolution block in the last layer of the decoding path to obtain more discriminative features. Moreover, the residual mechanism was integrated into a spatial squeeze-and-excitation layer and convolutional block attention module, which improved sensitivity and prediction accuracy. A mixed loss integrating binary cross-entropy and Jaccard loss was used to ensure more balanced segmentation. The proposed method was applied on CRACK500 image database, and achieved a superior performance with Dice, Jaccard, and accuracy values of 79.83%, 69.92%, and 96.94%, respectively. The quantitative and qualitative experimental results show that our method achieves high-performance road crack segmentation and can adapt to cracks with complex background and more interference.

Open-air optical flow estimation is crucial for autonomous driving and traffic monitoring. However, it is still challenging to obtain high-precision optical flow under severe imaging environments. To alleviate this problem, we propose a self-adaptive optical flow estimation framework for harsh outdoor scenes, which consists of a frame-adaptive learnable module (FAL-module), a streak-invariant feature extractor (SFE), and a feature matching and flow propagation network. In the framework, three components are jointly learned in an end-to-end manner. The input frames can be adaptively enhanced without sacrificing optical flow-related information by FAL-module. And the SFE decreases the influence of streak-like noise generated by rainy and snowy. Compared with the state-of-the-art methods, the experimental results show that our approach can significantly improve the optical flow estimation quality in multiple harsh outdoor scenes without performance penalties in normal scenes.

In the last decade, the success achieved by convolutional neural networks (CNN) has changed the trend of research in the computer vision field from manual features to learned features for various applications. However, designing the architecture of CNN and selecting optimum values of its hyperparameters poses a critical challenge because of their massive and complex search space. Automatically picking up the optimal values of hyperparameters may facilitate the designers of CNN-based deep learning models. Swarm intelligence (SI)-based improved competitive gray wolf optimizer (ICGWO) in continuous search space is proposed for optimizing hyperparameters of custom-designed CNN architecture. To evaluate the performance of the ICGWO, we employ 16 benchmark functions. We also assess the effectiveness of the ICGWO by evaluating it with six state-of-art meta-heuristic algorithms. The experiment results show that the ICGWO can obtain the best accuracy on eleven benchmark functions and second best on two benchmark functions. The optimized CNN recognizes static signs of English alphabets in Indian Sign Language (ISL). We also evaluated the optimized CNN on the publicly available ISL dataset for the English alphabet, where it attains 99.93% validation accuracy. In order to evaluate the efficacy of the proposed variant of CGWO, we optimized the hyperparameters of CNN using the existing SI-based search optimization algorithms namely particle swarm optimization, whale optimization algorithm (WOA), differential evolution, and gray wolf optimizer as well.

Traffic sign recognition plays a crucial role in the development of automated transportation systems, and its accuracy and real-time capabilities are fundamental to ensuring automatic driving safety. However, the size of most traffic signs is <0.5 % of the traffic scene, and the imbalanced sample distribution can limit the accuracy of recognition models. To address these problems, we propose a small-scale sensitive traffic sign detection algorithm, named GH-YOLOV5. Specifically, we present a gated enhanced module to enhance the expression of small targets, which uses a two-dimensional mask to extract the position features from the bottom to the top. We also design a hierarchical context embedding-transformer module to correctly migrate the cross-layer features of small targets to deep semantic features. Moreover, to solve the problem of extremely unbalanced categories, we put forward a new data enhancement strategy, which uses the spatial density distribution of traffic signs to synthesize samples for a small number of classes. Finally, we use a dynamic resolution strategy to compensate for small target information. The experimental results are conducted on the widely used TT100k dataset, and our results show that the proposed algorithm significantly improves the model’s ability to perceive small targets and achieves values of 90.83%, 90.32%, and 94.33% for precision, recall, and mean average precision, respectively. Therefore, we provide an efficient traffic sign recognition scheme for advanced driver assistance systems.

Most existing studies on person reidentification (Re-ID) utilize deep feature representation learning. However, images often contain occlusion situations and nondiscriminative personal information. To extract more representative features, some researchers extract implicit deep semantic information by designing complex modules, such as mask maps and human pose landmarks. However, this can introduce complex human annotation and computational work. To overcome these issues, we propose a Re-ID model called multifeature fusion network (MFFNet). Our network does not require any additional auxiliary information and incorporates two new designs: the feature refinement pooling block (FRPB) and the feed-forward conduction structure (FCS). Based on the “split-learn-merge” principle, the FRPB decomposes a person’s features into coarse-grained to fine-grained representations. The FRPB learns the corresponding local detail information and merges the multigranular features into the partial person representation. To address the issue of most current methods heavily relying on accurate bounding boxes, the FCS enables character matching at different resolution scales by learning multiple semantic levels of representation. Through a series of ablation experiments, we demonstrate that the proposed strategy is effective for person Re-ID tasks. The results indicate that MFFNet achieves more competitive experimental results than the existing state-of-the-art methods.

The semantic segmentation of point clouds has achieved significant progress in recent years. However, most current methods rely on complex stacking of modules to learn local features, resulting in a complicated network structure. We propose an adaptive offset self-attention network (AOSANet), a simple and efficient point cloud learning framework for classification and segmentation tasks. AOSANet has permutation invariance when processing three-dimensional point cloud data and can dynamically learn local features based on the input features. To better fuse the local features, we introduce the maximum feature extraction module, which performs maximum feature weighting prior to local feature fusion. This allows the network to concentrate on the most important features and improve overall robustness. Through extensive experiments on publicly available datasets, the proposed method achieves a state-of-the-art performance on point cloud classification and segmentation tasks.

Mudras in traditional Indian dance forms convey meaningful information when performed by an artist. The subtle changes between the different mudras in a dance form render automatic identification challenging as compared to conventional hand gesture identification, where the gestures are uniquely distinct from each other. Therefore, the objective of this study is to build a classifier model for the identification of the asamyukta mudra of bharatanatyam, one of the most popular classical dance forms in India. The first part of the paper provides a comprehensive review of the issues present in bharatanatyam mudra identification and the various studies conducted on the automatic classification of mudras. Based on this review, we observe that the unavailability of a large mudra corpus is a major challenge in mudra identification. Therefore, the second part of the paper focuses on the development of a relatively large database of mudra images consisting of 29 asamyukta mudras prevalent in bharatanatyam, which is obtained by incorporating different variabilities, such as subject, artist type (amateur or professional), and orientation. The mudra image database so developed is made available for academic research purposes. The final part of this paper describes the development of a convolutional neural network (CNN)-based automatic mudra identification system. Multistyle training of mudra classes on a conventional CNN showed a 92% correct identification rate. Based on the “eigenface” projection used in face recognition, “eigenmudras” projections of mudra images are proposed for improving the CNN-based mudra identification. Although the CNNs trained on the eigenmudra-projected images provide nearly equal identification rates as that obtained using the CNNs trained on raw mudra grayscale images, both models provide complementary mudra class information. The presence of complementary class information is confirmed by the improvement in the mudra identification performance when the CNN models trained from the raw mudra and eigenmudra-projected images are combined by computing the average of the scores obtained in the final softmax layers of both models. The same trend of improved mudra identification is observed upon combination of the average score level of VGG19 CNN models of the raw mudra images and corresponding eigenmudra-projected images.

Compared with typically developed people, patients with autism spectrum disorder (ASD) have many differences, including visual attention. Recently, researchers have employed saliency prediction to conduct early diagnosis for patients with ASD by observing whether the fixation maps of the patients are consistent with prediction results. Though numerous saliency models have been designed, which promotes the performance of saliency prediction significantly, the efforts major in atypical attention prediction are fewer. Therefore, we propose a novel atypical saliency model, namely attention-guided dual-branch integration network, to perform atypical visual saliency prediction. Generally, our model is a U-shaped architecture including an encoder and a decoder. In the encoder, we first deploy the dual-branch fusion module to extract multiscale deep features, where the dual branches consist of a coarse branch and a fine branch. Particularly, the multiscale features of each branch are integrated in a bilateral way. Then, we employ the multiattention module consisting of dilated convolution, channel attention, and spatial attention to further elevate the high-level semantic cues. Particularly, the attention units are constructed in a cascaded way. Besides, to give an effective representation for the differences between atypical saliency estimation and the typical saliency estimation, we design a new discriminative mask to compute the loss function. Extensive experiments are performed on challenging datasets, and the results show the superiority and effectiveness of our atypical saliency model when compared with the state-of-the-art saliency prediction models.