The current automatic optical detection technology is challenged by the following two aspects: it is difficult to obtain enough defect samples, and the types are extremely unbalanced; the appearance defects are diverse and complex. The above problems seriously affect the detection accuracy and efficiency of the appearance defects of polarizers. Considering these issues, a new depth antagonism method of anomaly detection without real defect samples is proposed. An encoder is used to capture the regular characteristics of the stripe-structured light defect image and a decoder is used to reconstruct the defect-free image. An encoder module is then used to form an unsupervised countermeasure network. Finally, the abnormal score is calculated according to the difference between the reconstructed image and the sample image. In the training phase, synthetic defects are added, and the target potential loss function is improved to further increase the detection accuracy. The experimental results for a polarizer appearance defect data set-considering factors such as light imbalance, noise, and camera distortion-show that the area under curve of the test results of the proposed method reaches 97.9%, the average detection time of a single image is 19.2 ms, and the detection accuracy is 94.6%, which is superior to other methods such as GANomaly. The effectiveness and robustness of the proposed method are verified.