The demand to design lightweight semantic segmentation models on mobile devices is growing. Current U-shape structures can improve the segmentation accuracy. However, they can hardly achieve lightweight requirements due to their inefficient encoders. Besides, partial details and edges are damaged during the process of repeated downsampling. To this end, we propose an efficient and lightweight U-shape network (ELUNet) for real-time semantic segmentation. In the encoder, a light split-shuffle convolution block is designed as the key component of feature extraction to achieve high-precision segmentation in the resource-limited scene. Furthermore, we propose a bridge channel attention module in the skip connection to selectively emphasize the valuable features. In the decoder, we propose an upsample feature fusion module to capture global contextual information, significantly improving the ability of the network to extract spatial information. Moreover, we design an edge refinement module to refine the segmentation predictions further. Extensive experiments prove the effectiveness of the ELUNet on Cityscapes and Camvid benchmarks. Specifically, the ELUNet contains only 2.0M parameters and achieves 73.3% mIoU on Cityscapes validation set with the speed of 52.6 FPS for a 512  ×  1024 input image on a single 1080Ti GPU.