In order to effectively reduce the peak-to-average power ratio (PAPR) of orthogonal frequency-division multiplexing (OFDM) signals, an improved precoding algorithm based on square root Better-Than Nyquist pulse is proposed. A 60 GHz radio-over-fiber (RoF) system with precoded 5 Gb/s 4-order quadrature amplitude modulation (4QAM) OFDM downstream signals is demonstrated by simulation so as to evaluate the effectiveness of the proposed algorithm. The simulation results show that, compared with existing precoding method, the improved algorithm further improves the PAPR reduction performance by 0.3 dB. However, considering the balance between bandwidth sacrificed and PAPR reduction performance, the data redundancy no more than 20% is recommended. Meanwhile, the obtained maximum PAPR of the signals is independent of the number of subcarriers. Moreover, for both back to back and transmission cases, the receiver sensitivity can also be improved by 0.4 dBm by using the proposed precoding algorithm. Better bit error rate (BER) performance can be achieved by high-level modulation. Hence, it is suitable for future 60 GHz OFDM-RoF system applications.