In this study, a color image encryption algorithm is proposed based on the symmetry of Fourier transform and random multiresolution singular value decomposition (R-MRSVD). First, the average value of normalized plaintext images is considered to be the initial value of logistic-exponent-sine mapping. Thus, a random matrix and position index are generated. Then, a two-dimensional discrete Fourier transform is applied to each color channel. Because of the conjugate symmetry property of Fourier transforms, only half of the Fourier transform spectrum coefficients are preserved. Further, a real matrix is constructed by extracting the real and imaginary parts. Finally, R-MRSVD is performed, and the ciphertext image is obtained using Josephus scrambling operation. The pixel characteristic of the plaintext image is considered to be the initial value of the chaotic sequence; thus, the high sensitivity and security of the algorithm can be ensured. The real-valued ciphertext image is convenient for storage and transmission. The quality of the decrypted image, statistical characteristic, key sensitivity, chosen-plaintext attack, and robustness of the algorithms are verified. The simulation results demonstrate the feasibility and security of the proposed color image encryption algorithm.