To track and estimate the pose and position of known rigid objects efficiently in complex environment, a method coupled three dimensional (3D) particle filter (PF) framework with M-estimation optimization in a closed loop is proposed. A novel similarity observation model is constructed based on multilevel line representation; line correspondences between 3D model edges and two dimensional (2D) image line segments are received easily based on the tracking state of PF. After that, line correspondences are provided for M-estimation to optimize the pose and position of objects. The optimized particles are fused into the particle filter framework according to the importance sampling theory. Moreover, to speed up the proposed method, line detection and search space is limited in a local region of interest (ROI) predicted by PF. Experiments show that the proposed method can effectively track and accurately estimate the pose of freely moving objects in unconstrained environment. Comparisons on synthetic and real images demonstrate that proposed method greatly outperforms the state-of-art method in accuracy and efficiency.