Considering that the accuracy of target tracking and recognition is disturbed by the moving shadow in intelligent video surveillance, we proposed a shadow detection algorithm based on color and gradient histogram back-projection. On the basis of the moving object region detected by visual background extractor model, the shadow pixels were screened by the intensity and chromaticity in HSI color space at first. Then, the gradient histogram back-projection method was used to distinguish the moving object from the shadow. Finally, the detection results based on the above two methods were reasonably fused to get the final shadow area. The experimental results show that: Compared with the typical algorithms, the proposed method can effectively improve the accuracy and robustness of shadow detection. Therefore, it can be applied in accurate moving object detection and recognition.