In order to reduce the effect of polarization mismatch on detection of Brillouin scattering spectrum in the coherent detection system for Brillouin optical time-domain reflectometry (BOTDR), based on the feature of Brillouin scattering light and the principles of the polarization spreading and the polarization diversity, the time-domain polarization diversity (TDPD) technique is introduced. The TDPD is implemented easily and effectively for BOTDR. The theoretical analysis shows that with TDPD more than half of signal power is heterodyned with the local light no matter how their polarizations vary, and the received signals are insensitive to the random variation of the polarization of Brillouin backscattering light. The TDPD is implemented by performing simple and non-feedback polarization control on the local light, and the original architecture of BOTDR is less changed. The comparison between the system with and without TDPD shows that the fluctuation of the received signal light power with TDPD is 4.2 times less than that without TDPD.