Compared with visible light imaging technology, infrared optical imaging technology (see Fig. 1) is less affected by weather and illumination and is a potential auxiliary solution method for the future of the autonomous driving system. Based on the above characteristics, this study proposes the hold surrounding’s key (HSK)-you only look once (YOLOv7) algorithm. First, based on the characteristics of the infrared image, this study optimizes the network structure of YOLOv7 and proposes a vehicle and pedestrian detection algorithm based on the improved YOLOv7. Aiming at the problem that the reasoning speed and detection accuracy of pedestrian and vehicle detection algorithms based on infrared images are challenging to balance, occupy large storage space, and are difficult to deploy and run in real-time in low and medium-performance devices, the MPConv is added to replace the Conv structure in YOLOv7. In view of the false detection, missed detection, mutual occlusion and overlap of detected objects, and other situations that the YOLOv7 algorithm is prone to cause in the actual deployment environment easily, a tiny object detection layer is added. At the same time, to solve the problem that infrared optical imaging systems are prone to noise caused by external factors, this study introduces the TRPCA method for image denoising in the preprocessing process of the YOLOv7 algorithm. In the end, the HSK-YOLOv7 algorithm is verified using the self-made infrared traffic object detection dataset and the publicly available FLIR dataset to verify the detection effect of the HSK-YOLOv7 algorithm on near-infrared images and thermal infrared images. The parameter quantity of our algorithm is 37.3M, and the computing throughput is 107.5 GFLOPs. The detection speed on the self-made dataset and FLIR dataset reaches 163 frames per second (FPS) and 71.8 FPS, respectively, and the mAP@0.5 indicator reaches 94.08% and 61.3%, respectively. In general, HSK-YOLOv7 can meet the real-time requirements of the autonomous driving system while ensuring detection accuracy.