Aiming at the problem of monocular camera pose estimation under the matching relations of unknown feature points, a new camera pose estimation algorithm is proposed based on the Softassign algorithm. Based on the advantages of orthogonal iteration and Softassign algorithm, the collinear error of the object space in three-dimensional (3D) feature points and two-dimensional (2D) feature points is used as the basis for determining the matching relationship between feature points and the objective function value for calculating the camera pose. A iteratively way determines the matching relationship of feature points and the pose of the camera. It can handle the case where the 3D/2D feature points are in one-to-one correspondence, and the case which occlude 3D feature points and erroneous 2D features. The results of the synthetic image experiments show that the success rate of this algorithm is over 82% in one case with image noise, occlusion and clutter. The experiment of real image can verify the property of algorithm.