An image processing method for accurately calculating the local fringe direction of moiré patterns was proposed that can be applied to the moiré deflectogram of a spherical progressive addition ophthalmic lens. The empirical characteristics of a collected moiré pattern were used as the basis for analyzing misalignments between assumptions and actual measurement conditions during the derivation of the measurement theory. Further investigation shows that the moiré fringe angle required for actual measurement is based chiefly on the fringe phase field rather than the intensity field. Based on this, a method using phase fitting to obtain the local angle of fringes was proposed. The method first obtained the phase of the moiré fringe by phase shifting and next fitted the phase to the Zernike polynomial to obtain the polynomial expression of the phase. The method then calculated the fringe angle of the corresponding position by solving the partial derivative of the polynomial in each of two directions. Theoretical analysis and simulation results show that this method can eliminate the influence of the uneven distribution of background light and light source amplitude, making it insensitive to fringe noise and fringe period variation. Angle calculation accuracy can reach 0.2°.