2.1

Introduction to YOLOv5 algorithm

The Yolov5 model is separated into five versions based on the depth and width of the network, namely Yolov5x, Yolov5l, Yolov5m, Yolov5s, Yolov5n. Due to the high real-time performance requirements of the model for foreign object recognition on transmission line towers, this paper considers the model’s identification accuracy and inference speed, and chooses Yolov5s with a small model weight as the basic model for research.

The four key sections of Yolov5s are input, backbone, neck, and prediction. The input side adopts the Mosaic algorithm, and randomly selects four pictures in the data set to stitch together after cropping, zooming, flipping and other operations to increase the robustness of the model. Backbone integrates Focus, CSPNet, and SPPF modules. Focus slices the input image to lessen the amount of information lost during the downsampling process. The CSPNet structure divides the original input into two parts, one branch performs the Bottleneck×N operation, the other branch directly performs the convolution operation, and then performs the concat operation on the two branches to reduce quantity of channels, so that the BottleneckCSP’s output and input. The same size so that the model can learn features better. The SPPF module uses multiple small-size pooling kernel cascades to replace the SPP module contains a large-size pooling kernel to fuse feature maps of different receptive fields, improve feature expression capabilities and speed up operation. The Neck part is mainly composed of FPN and then PANet network. FPN constructs high-level semantic feature maps at all scales via top-down side connections. A bottom-up path is introduced in the PANet network to make the underlying features better. The transfer to the top layer to combine the features of different layers improves the utilization of low-level features. The bounding box loss function used by the prediction layer is CIOU loss, and redundant prediction boxes are filtered using the non-maximum suppression approach.

2.2

RepVGG module

Although complex multi-branch models such as ResNet can achieve high accuracy, they also have certain defects. This will lead to a decrease in the calculation speed of the model and a decrease in memory usage. Some nodes will increase memory consumption, have high requirements for equipment, and increase hardware overhead. RepVGG is an improved classification network based on the VGG network, as shown in Figure 2. Its main features are that the Identity and residual branches are added to the Block module of the VGG network, and in the model reasoning stage, 3×3、1 ×1 convolution branch, which facilitates the deployment of the model and speeds up the reasoning.

Figure 2.

RepVGG training model and inference model structure

2.3

Hybrid attention system

Introducing the convolutional neural network’s attention mechanism can reduce the negative impact of complex information when recognizing objects, that is, focus on key information and suppress irrelevant data to increase the model’s resilience and ability to recognize objects. CBAM is a lightweight attention mechanism module that combines space and channels, and can be integrated into the convolutional neural network architecture with only a small computational overhead. Input the feature map’s dimensions F∈R^{C× H×W} to the CBAM module, After the input feature map undergoes two different pooling average pooling and maximum pooling, the two one-dimensional tensors obtained by pooling are obtained by sharing the layer that is totally connected and the corresponding activation function to obtain the correlation between channels, and the two output Merge to get the weight of each channel with features to generate channel attention features M_c∈R^1×1×C, The mathematical expression of channel attention Mc is seen in formula (1).

The spatial attention module receives its input from the channel attention module’s output, or Mc, and the procedures of global maximum pooling and global average pooling are respectively performed along the channel dimension to obtain a two-dimensional feature map F_avg∈R^{1×H ×W}, F_max∈R^1×H×W and perform dimension splicing on the channel to obtain an efficient feature description, perform convolution operation on the concatenated results, and output the map of spatial attention Ms after the Sigmoid activation function, The mathematical expression of channel attention Mc is seen in formula (2).

2.4

Improved non-maximum suppression

A crucial step in the target detection process is non-maximum suppression (NMS). The primary concept is to rank the candidate boxes by confidence scores from high to low, and choose the candidate box with the most assurance, and the remaining candidate boxes. Perform IoU comparison and filter out boxes that exceed a set threshold of IoU. The NMS calculation formula(3) is as follows, si represents the confidence score of the i candidate frame, M is the candidate frame with the highest confidence score, Nt is the set IoU threshold, and bi is the rest of the candidate frames. The most issue with NMS is to forcibly remove all candidate boxes whose IoU exceeds the threshold. The detection of an object will be unsuccessful if it occurs in the overlapping area, resulting in missed detection, and reduce the recall rate of the model.

In the scene of a real transmission line, foreign objects on the transmission line are often irregular in shape and easily form mutual occlusion with the tower. For this problem, it is a more effective method to use the confidence decay function to adjust the overlapping frame with the candidate frame with the highest confidence. The Soft-NMS algorithm is different from NMS, which regards all overlapping candidate frames as redundant. Soft-NMS believes that there may be effective detection targets even in candidate frames with more overlaps, but the larger the overlapping area, the lower the possibility. Therefore, in the algorithm execution, instead of directly deleting the frame whose IoU is greater than the threshold, the connection between IoU and the confidence score is first established. The formula is as follows, and the Gaussian penalty function is used to reduce the confidence score of the candidate frame until it is lower than the IoU threshold. Avoid deleting the correctly positioned candidate boxes in the case of overlapping detection targets, thereby improving the detection accuracy and recall rate of the model. In the formula, D is the set of all valid candidate frames retained. When IoU is 0, the confidence score remains unchanged. When 0<iou<1, the confidence score decays. Figure 3 displays the Soft-nms algorithm’s pseudocode.

Figure 3.

Soft-nms algorithm pseudo code

2.5

Add detection layer

The initial Yolov5 model has three detection frames, and the output sizes are 80×80, 40×40, 20×20, which are used for the recognition of small, medium and large targets respectively. However, foreign objects on the transmission line are often of different scales, and The background is complex and easily occluded, and the shallow features contain more details and spatial information. Therefore, a shallow detection layer with a scale of 160×160 is added to increase the range of network detection scales as shown in Figure 4 and improve the model’s the capacity to recognize objects of various sizes,detection performance.

Figure 4.

Add detection layer

3.1

Dataset and model training

Since there is no public data set for foreign objects on transmission lines, the data sets in this experiment are 3726 private data sets constructed for three types of foreign objects: bird’s nest, kite, and balloon. Since there aren’t enough training samples, the data set needs to be expanded. In deep learning, training samples are expanded. Common methods for augmenting datasets include cropping, flipping and rotating, scaling, shifting, adding Gaussian noise and color dithering, etc. In this paper, Through the use of techniques like rotation, shifting, and cropping, the data set is increased to 6500, and the training set to the verification set to the test set ratio is 8:1:1.

If the model is trained from scratch, the effect will not be ideal, especially if the quantity of the training sample is insufficient. The ImageNet dataset has been widely used as a pre-training dataset because its data samples are large enough and more than 1,000 categories contained in the dataset come from daily life, making it suitable for training universal models. Therefore, in order to avoid overfitting due to insufficient training samples, this paper first pre-trains on the ImageNet dataset through transfer learning to obtain pre-training weights, so as to avoid the backbone feature network weights from being too random. First freeze the weights in the first 169 layers of the model backbone network, and train the remaining layers. At this time, the network for feature extraction won’t change. Unfreeze the previously frozen layers for training, fine-tune the weight parameters of all layers, and finally the trained model can be used for foreign object recognition on iron towers. Since the learning rate and batch size of the freezing and unfreezing stages are different, as well as the optimizer used, Table 1 displays the experimental parameter.

Table 1.

Experimental parameter

3.2

Experiment configuration and evaluation indicators

The experimental platform’s hardware environment, as designed in this paper uses the CPU as Interl Core i7-11700K, the GPU as NVIDIA GeForce RTX3070TI, and the memory as 32GB. The deep learning framework is the foundation of the software environment pytorch1.8.1 and python3.9. To evaluate the model’s effectiveness, different target detection algorithms are used on the same data set to compare with this paper’s algorithm, and the accuracy, average precision, mAP and model detection speed FPS commonly used in target detection are used as important evaluation indicators of the model. P_re is used to measure the likelihood that all samples predicted by the model to be positive are in fact positive. Precision rate and recall rate added together form the area under the P-R curve or AP, which is used to measure the quality of the recognition precision of the model on a single category. mAP represents the average AP of non-category samples, and the number of frames the model recognizes per second is referred to as FPS. TP stands for the proportion of positive samples deemed positive, FP for the proportion of negative samples deemed positive, and FN for the proportion of positive samples deemed negative.

3.3

Analysis of results

In order to gauge how the model’s performance would be affected by the improvement measures described in this paper, the improved results are analyzed by means of ablation experiments. The test results are displayed in Table 2. The RepVGG module increased the mAP value from 76.5% to 77.2%, indicating that the introduction of the RepVGG module can improve the model’s capacity to extract features.Compared with the second group of experiments and the third group of experiments, after adding the CBAM module, the mAP value increased by 1.2%, indicating that the model combined The attention mechanism module can suppress the influence of other negative information and strengthen the network’s capacity to express features. Comparing the third group of experiments with the fourth group of experiments, the mAP value increased to 79.1% after using Soft-NMS instead of the conventional NMS algorithm, indicating that Soft-NMS can improve the recall rate of the model when similar objects overlap. Comparing the fourth group of experiments with the fifth group of experiments, the mAP value increased from 79.1% to 80.6% once the detecting layer has been added. The results showed that the accuracy of the model’s detection of foreign objects of different scales was improved and the rate of missed detection was reduced.

Table 2.

Ablation Experiment Results

The technique in this work is contrasted with various popular target detection models that are currently in use, such as Faster-RCNN, SSD, Yolov4 and Yolov5, to verify the advantages of the improved Yolov5 model in the detection of foreign objects on transmission lines. The comparison figures are displayed in Table 3.The improved algorithm proposed in this paper has a mAP value of 80.6% in foreign object detection on transmission lines, which is 4.1%, 5.9%, 11%, and 5.5% higher than Yolov5, Yolov4, SSD, and Faster-RCNN respectively. The model of two-stage detection Faster-RCNN Although the detection accuracy is high, the size of the model is too large, which is not suitable for carrying in mobile devices. Compared with other single-stage models in terms of model size, the improved model has a large increase in size compared to the Yolov5 model by 17.7MB. reduce.

Table 3.

Comparison of the results of several algorithms’ recognition