The problem of optical distortion caused by the contamination of the toothed gear profile surface was studied which affects the accuracy of measurement on the toothed gear profile deviation and the pitch deviation. Iterative Approximation-Proximity Discrimination algorithm(IAPD) using visual measurement is proposed to determine the image edge distortion of the toothed gear profile. The algorithm constructs the mapping relationship between the normal pixel offset in the edge transition zone of the involute profile image and the polar diameter of the pixel. This mapping converts the complex two-dimensional image edge signal into a one-dimensional signal, which is easier to process. The wavelet denoising algorithm was used to process the signal to extract the distortion features of the toothed gear profile edge, and the variable threshold iterative fitting algorithm was used to separate the toothed gear profile tilt deviation. According to the reasonable range of the normal toothed gear profile error, K-proximity classification method is adopted. The starting and ending position of the edge distortion of the toothed gear profile image is determined, and the positioning basis is provided for correcting the edge distortion of the toothed gear profile image during the measurement of the pitch and the toothed gear profile deviation. In order to verify the reliability of the IAPD algorithm, according to the similarity of the real edges of adjacent ipsilateral tooth profiles, the sub-pixel edges are extracted from the distortion tooth profile and the distortion-free tooth profile image, and similarity comparison is performed to realize the profile edge similarity comparison algorithm of distortion region recognition. The algorithm is used to verify the distortion region discrimination accuracy of the IAPD algorithm. The experimental results show that the toothed gear profile image edge distortion detecting algorithm can identify the image distortion region with good accuracy. The accuracy of radial range of the distortion region can reach 2.5 pixels(50 μm), which can meet the requirement of the image edge distortion correction. In addition, the algorithm is easy to understand and highly effective, which is easy to deploy and can meet the requirements of real-time calculation during gear measurement.