For the shortcomings of correlation delay shift keying chaotic communication system, mutual interference between the adjacent bits leads to high bit error rate, a Hadamard code correlation delay shift keying chaotic communication scheme based on quantum key distribution protocol is proposed. In the system transmitter, the chaotic signal through Hadamard function for spreading spectrum is used as reference signal, and the sent signal is a sum of the reference signal, delay signal and data modulation signal, and the system delay time is determined by quantum protocol distribution. Simulation results show that because of the orthogonally of Hadamard function, the proposed system can well eliminate the mutual interference between adjacent bits and keep the advantages of original correlation delay shift keying, such as chaotic communication scheme, reducing system bit error rate. Moreover, the proposed system has quantum key distribution protocol with high security.