Capsule network (CapsNet) is well known as an evolution of classical convolution neural network, which is good at recognizing postures, orientations, and textures, thus achieving promising results in areas such as image classification. However, the high computational complexity hinders them from building larger models. Fortunately, reducing the complexity of the routing of CapsNet through the channel attention mechanism has achieved significantly good results, whereas there is a possibility of further optimization of the storage and precision. Therefore, we propose an improved version of combination of multiscale routing and self-attention mechanism on the basis of the traditional channel attention routing. First, we absorb multiscale information into the routing process, thus enriching the representation capability of the capsule. Second, the global information of the capsule is captured by the self-attention routing, which leads to less misclassification errors. Finally, the proposed self-attention routing is further improved by a soft-threshold strategy, which resists the interference of background noises on complex datasets. Experimental results on MNIST, affNIST, and Cifar10 show that the proposed methods can obtain better performances with fewer training parameters compared with traditional methods on average.