To enhance the complexity of the output sequence of the optical injection structure and improve the security of the bit plane encryption, this work proposes an image encryption algorithm based on optical chaos and quadrature amplitude modulation (QAM)-like. The external disturbance generated by the mutual injection of the dual-drive laser improves the complexity of the optical chaos output from the laser in the optical injection structure. In the image encryption algorithm, this work adopts a QAM-like coding mode, which encrypts the pixel value of the plaintext image by transforming the coordinates of the QAM constellation diagram and the coding. In the coding stage, the different coding modes of QAM-like are selected using the encryption sequence composed of optical chaotic sequence and key to transform the color plaintext image into the coded image, and the pixel value of the plaintext image is changed. In the scrambling stage, the encrypted sequence is used to scramble the multi-dimensional rows and columns of the three-dimensional coded image, and the position information of the pixel points is encrypted. In the diffusion stage, the scrambled images of each color component are diffused by GF(17) multiplication and finally merged to generate ciphertext images. Simulation results show that the proposed image encryption algorithm has sufficient key space and information entropy is close to the theoretical maximum, which can effectively resist known plaintext attacks, selective plaintext attacks, and statistical feature attacks.