In order to explore the influence of the modulation depth in subcarrier OFDM optical transmission system, the bit error ratio response of subcarrier OFDM optical systems in intensity modulator (IM) is investigated by simulation and experiment when the modulation depth is changing. Simulation and experimental results show that there is an optimum carrier sideband ratio (CSR), which is about 20, when subcarrier OFDM signal transports through optical fiber. Too large or too small CSR will cause the system′s bit error ratio performance decrease. It is shown that the PM system has a good transmission characteristics when the sidebands is suitable and constant with the modulation depth.