An optical quantum random number generator based on the time randomness of single-photon pulse is proposed. A constant-intensity light source attenuating into single-photon state and a single photon detector are used to generate single-photon random pulses. The random bits are extracted by continuously comparing the time intervals between two adjacent pulses in the single-photon random pulses sequence. A random number generation rate of more than 10 M bit/s is obtained by designing high-speed single-photon detector based on micro-channel plate and field programmable gate arry (FPGA) based random bit extraction circuit. In order to reduce the correlation coefficient of random bit sequence, measurement accuracy of the time interval is improved by using a constant fraction discriminator and a frequency-multiplied counting clock. Correlation coefficient of the random bit sequence is less than 0.001, when the random bit generation rate is less than 10 k bit/s. The random bit sequences are tested by random number test program ENT and DIEHARD. The test results show that random bit sequences have good randomness, do not require post-processing and fully meet the standards of true random numbers.