The absorption and scattering of light in seawater channel cause signal attenuation, and the turbulence of seawater causes signal amplitude fluctuation, both of which will reduce the bit error rate (BER) performance of underwater wireless optical communication (UWOC) system. The effects of the two channel characteristics on the signal performance were considered comprehensively, and a method was proposed to equate the transmission distance and turbulence probability density function to the system signal-to-noise ratio (SNR) and turbulence noise, and then the signal attenuation and turbulence noise were combined into the signal waveform to establish the underwater composite channel signal transmission model. According to the experimental system parameters, the signal transmission waveforms of Gaussian minimum frequency shift keying (GMSK) modulation under composite channel were simulated, and the one-bit difference demodulation algorithm was used to compare the demodulated waveforms with the original waveform, and the influence relationships of composite channel on the system BER performance was analyzed. The simulation experiment results show that, compared with on-off keying modulation (OOK), pulse position modulation (PPM), GMSK system can obtain the SNR gain of 3.3 dB, 4.8 dB respectively only in the attenuation channel with seawater attenuation coefficient of 0.151 m^-1. Under the composite channel, GMSK modulation performance is superior to OOK modulation and PPM modulation. When the water attenuation coefficient is 0.151 m^-1, and turbulence intensity variance is smaller than 0.16, GMSK modulation system has no error rate limit, the system BER is decided by signal attenuation and turbulence noise and Gaussian noise together, GMSK modulation achieves SNR gain of 4.35 dB compared with PPM modulation. Furthermore, turbulence intensity variance is greater than 0.16, system BER arrives limit, which value is determined by the turbulence intensity, and the limit value of BER increases nonlinearly with the increase of turbulence intensity.