To extract a smooth and accurate strain field from a noisy displacement field obtained by digital image correlation method, the moving least-square (MLS) fitting method is often adopted. However, as the MLS fitting method is computationally expensive and unstable when applied to large datasets, the pointwise moving least-square (PMLS) fitting method is proposed herein to improve the computational efficiency and stability. The feasibility and accuracy of the strain field fitted by the PMLS fitting method were explored on simulation experiment under two conditions of continuous displacement field and discontinuous displacement field, and then the strain field of the two groups of measured data images of the continuous displacement field experiment and the crack experiment were obtained. The PMLS fitting method was compared with the classical point-by-point least-square (PLS) method and the MLS fitting method. The results show that the accuracy of the proposed method is effectively improved compared to the PLS method by more than 16% when a large window is used, the computational efficiency increases by more than 27 times compared to the MLS method, and the stability of the MLS method is significantly enhanced. For the discontinuous displacement field, the problem of strain distortion in the discontinuity regions can effectively be addressed, which confirms the applicability of the proposed method to discontinuous displacement fields.