Extended Kalman filter (EKF) is a popular method to address the fast rotation of the state of polarization and large polarization mode dispersion in coherent optical communication. But it is usually underperforming in high-order modulation. Radius-directed idea is introduced to enhance the performance of EKF-based method. We call the proposed scheme radius-directed extended Kalman filter (RD-EKF). The performance of the proposed Kalman scheme is verified both in the polarization-division multiplexed 16QAM and 64QAM system. By the results, the RD-EKF scheme has better performance and less complexity than conventional EKF method in the high modulation format system.