Carrierless amplitude and phase (CAP) modulation is a way of digital quadrature amplitude modulation (QAM). As the CAP modulation has the advantages of high spectrum efficiency, low cost, less complexity, etc, it gradually become an alternative option for the implementation of short distance high speed optical communication system. Firstly, the properties of these two kinds of modulation mode were analyzed by comparing the power spectrum of CAP modulation and QAM, then under the conditions of atmospheric channel simulation model(Gamma-Gamma distribution of light intensity fluctuation model), the influences of rolling pressure coefficient of filter, the length of filter and sampling clock precision on the performance of the CAP modulation wireless optical communication system were analyzed based on the Matlab simulation and the optimal parameter values were given. The simulation result shows that the sampling clock offset will cause the phase shift of the received signal, so it is needed to compensate the phase distortion in the receiving end. In addition, the research on the equalization algorithm needed by later was also carried out and it is hoped that it can provide some help for the design of CAP system in the future. Finally the simulation results had been verified by experiment.