The wide-field space camera with off-axis and three-reflection structure had large optical distortion, which led to the image motion blur during push-broom imaging of the plane-array detector with time delay integration (TDI) technology. According to the principle of TDI imaging degradation caused by distortion, the distortded image motion blur was converted into non-uniform motion blur, and the initial blur kernel was solved by fitting the image motion path, which was used as prior information to establish a semi-blind restoration model to further refine the blur kernel. Using the rough image edge restored by the initial blur kernel to guide the refinement, a multi-directional weight-heterogeneous total variation model was proposed to extract image structure information. In order to enhance the constraints of prior information on the refinement of the blur kernel, a regular term containing the initial blur kernel was constructed, so that the estimation of the blur kernel did not depend excessively on the image content, and a multi-scale iterative method was used to solve the problem. Finally, a regularization-constrained non-blind deconvolution method was used to remove image blur. Experimental results show that the compared with several existing deblurring algorithms, the deblurring effect of the proposed method is not only clear and natural, but also more stable in the estimation of the blur kernel of different sample images.