In the spin exchange relaxation free (SERF) atomic magnetometer, it is necessary to detect an extreme small optical rotation angle. Among various methods for detecting the rotation angle, the polarization modulation technique based on a photoelastic modulator (PEM) is preferred because it features lower noise and better stability at long-time scales. However, the output signal of the photoelastic modulator contains much noise and high-order harmonics, which seriously affects the performance of the atomic magnetometer. The principle of polarization modulation technique based on photoelastic modulator and the characteristics of the detected signal were analyzed. And a dual channel digital lock-in amplifier to detect the weak signal from the atomic magnetometer was proposed. This method simplified the lock-in algorithm, reduced the complexity of the circuit and accurately detected the amplitudes of the first harmonic and the second harmonic. Theoretical analysis and simulation results show that the detection system works well and detects weak signals accurately, and the simulation error is less than 0.1%.