There recently has been increasing interest in the research and application of the rotational Doppler effect (RDE), which paves a promising way to detect rotating objects remotely. In order to obtain more information about the rotating object from the rotational Doppler signal, composite vortex beams by coaxial superposition of orbital angular momentum (OAM) modes are often used as the probe beam. However, to the best of our knowledge, the RDE of composite vortex beams with arbitrary OAM spectra has not yet been comprehensively studied. In this paper, the correspondence between the OAM spectrum of a probe beam and the frequency spectrum of a rotational Doppler signal is theoretically analyzed. It is explicitly revealed that the RDE frequency spectrum of scattered light is related to the product of two autocorrelation functions: one from the OAM spectrum of probe beam and the other from the spiral spectrum of rotating object. On the basis of this relation, one can regulate the RDE frequency spectrum on demand via tailoring the OAM spectrum of the probe beam. As a proof of concept we design a special composite vortex beam to eliminate the broadening of the RDE spectrum induced by misalignment. These findings are of practical value in applications such as remote sensing and optical metrology.