Combined with the discrete sine transform (DST) and discrete cosine transform (DCT), a novel DST/DCT direct current bias-discrete multitone(DC-DMT) modulation scheme is proposed. This method is able to expand the number of independent subcarriers without increasing the complexity of system, and it is expected to improve the transmission rate of visible light system. The bit error rate (BER) expression of the modulation scheme and the complementary cumulative distribution of peak-to-average power ratio(PAPR) are derived. The effects of clipping noise on the BER performance of DMT, direct current-biased optical-orthogonal frequency division multiplexing (DCO-OFDM) and asymmetrically clipped optical-orthogonal frequency division multiplexing (ACO-OFDM) visible light systems are analyzed. The results show that the theoretical BER is consistent with the simulation results, and the BER performance of the proposed scheme is better at the same transmission rate compared with that of other methods. The distribution of PAPR is independent of the modulation order but increases with subcarrier number N. When N is larger, the theoretical conclusions match the simulation results. At the same transmission rate, the simulation results show that the BER performances of DST/DCT DC-DMT and DCO-OFDM are similar in low signal-to-noise ratio (SNR), which are better than those of PAM-DMT, DC-DMT and ACO-OFDM; however, the BER performance degrades in high SNR due to the clipping noise.