When a polarized camera is used to realize instantaneous phase-shifting and obtain photoelastic fringes, quantitative error and compensation analysis is often ignored. The error of the instantaneous photoelastic phase-shifting method in a circularly polarized field is primarily caused by a mismatch between the wavelength of the quarter-wave plate and the light source. Based on the Stokes vector and Mueller matrix, this paper theoretically derives the error and compensation equations of the photoelastic properties, and discusses the error distribution of the stress direction angle and phase retardation of samples. The results show that the isochromatic fringe error is directly proportional to the mismatch value. In addition, the isoclinic fringe error is nonlinear, but it is in direct proportion to the cosine of the mismatch value and inverse proportion to the difference between two light intensities. Furthermore, the paper analyzes the influence of the spectral width of the light source on the compensation effect, and finds that the spectral width is directly proportional to the wavelength and inversely proportional to the mismatch error. Finally, the correctness of the error analysis and compensation method is verified through the simulations and experiments of the diameter-stressed disk, and the isoclinic separation in the stress direction is discussed.