This paper proposes an asymmetric optical color-image encryption system based on vector decomposition and chaotic random phase mask. The introduction of vector decomposition makes the first phase mask also act as the main secret key when the input image is a real-value image, which makes the encryption system asymmetric. The two chaotic phase masks generated by the 2-D Henon chaotic map replace double random phase masks. The chaotic system’s initial values and control parameters were used as secret keys to increase the key space and provide more security for the encryption system. The following parameters were tested: key sensitivity, correlation coefficients of two adjacent pixels, occlusion attack resistance, noise attack resistance, and chosen-plaintext attack resistance of the encryption system. Test results validate the feasibility and security of the proposed encryption system.