Road damage detection plays a vital role in road safety maintenance. Minor road damage cannot often be identified efficiently because of the challenge of extracting damage information, which frequently lacks minor information. In addition, due to the irregular shape of alligator road damage, the performance of existing damage detection algorithms is seriously reduced. To solve the above problems, we propose an improved YOLOv5 model called YOLOv5-MCD for road damage detection. First, a multiscale dilated convolution module combined with an attention mechanism is proposed and used to replace the spatial pyramid pooling fast module in the neck of the original YOLOv5 model. Multiscale dilated convolution with a larger receptive field can be applied to obtain more damage feature information at different scales, and the attention mechanism aggregates the spatial and channel feature information of road damage images, allowing the model to pay more attention to damaged areas. Second, the cross-layer feature fusion method is proposed to merge more road damage features to enhance the original damage feature information. Finally, deformable convolution is used to improve the adaptability and performance of our method for complex alligator damage. Our method was tested and validated by the China-MotorBike RDD2022, and the F1-score and mAP@0.5 of our model were 93.5% and 95.5%, respectively. The experimental results show that our method can efficiently identify minor and alligator road damage and achieve better detection performance.