For the improvement of security capability of existing optical security system, a novel method for encrypting optical image using encoding on both space domain and fractional Fourier domain is presented. In this paper, image is Fresnel transformed and Fractional Fourier transformed successively, and phase encoding is implemented in both Fresnel domain and Fractional Fourier domain. The parameters in two transform can be designed as keys, and the wavelength-key can also be introduced into the encryption process successfully, then the keys of the security system are set on different domains, which can improve the security capability of the system. Computer simulation indicates that the method proposed in this paper can encrypt and decrypt image successfully, and the decrypted results are very sensitive to the keys, people unauthorized can hardly decrypt the image correctly without knowing them.