The phase shifting shadow moiré is an efficient technique for three-dimensional (3D)object surface measurement, of which the performance can be improved further. This paper focuses on improving its measurement accuracy without adding the complication of the experimental set-up. An effective method, based on the idea of random phase shifting, is proposed. In the method, the parameters of shadow moiré system are calibrated by stereo vision technique and the phase of the fringe pattern is extracted by three-frame random phase shifting algorithm. We assume that the introduced phase shifting is unequal and unknown, so the stringent requirements for the shifter are relaxed. The proposed method can lower the application requirement of shadow moiré. Besides, the proposed method is not affected by the background light in the phase demodulation process, and is insensitive to the non-sinusoidal light intensity distribution of the fringe pattern, and has the advantages of high accuracy and easy application. Experiments show that the proposed method has high accuracy and high speed and is superior to the existing typical algorithms.