We present the design of a novel bi-directional millimeter-wave radio-over-fiber (mm-RoF) system based on the millimeter-wave generation by optical frequency multiplication (OFM). A dual-drive Mach-Zehnder modulator is used to generate high-order optical side-modes which beat in the photo-detector, producing a 40-GHz carrier. Over 100-Mb/s orthogonal frequency division multiplexing (OFDM) modulation scheme is employed. The emphasis is on developing a mathematical model for optimizing optical modulation index to the Mach-Zehnder intensity modulator (IM) for OFDM signal with high peak-to-average power ratio which imposes a limitation on the system bit error rate (BER) performance due to the non-linearity of IM. The theoretical analysis on composite carrier to composite triple beat ratio is performed based on which extension to the system BER formula for quadrature phase shift keying/ multiple quadrature amplitude modulation (QPSK/MQAM) format is presented. The experimental proof is given in a 40-GHz RoF system at a bit rate of up to 280 Mb/s in 100-MHz bandwidth.