Based on traditional ground target recognition systems, a new ground target recognition system based on multi-sensor information fusion, using visible light sensors and infrared sensors, is developed in order to get effective target information in the cases of night, rain, snow and smog. Firstly, Gaussian template mean filter is adopted to the visible light image, and median filter is adopted to the infrared image. And then, the inter frame difference method and background difference method are combined to realize the detection of the moving target. An algorithm based on the coordinates of the center of gravity connected domain mark is presented and the target area is located successfully. Secondly, the Haar-like rectangular features are used to represent the vehicle, the original characteristics library is expanded and rotation single rectangle features are added to describe the vehicle shadow region. Finally, the traditional AdaBoost algorithm is improved and the visible light and infrared vehicle classifier is trained successfully. Experimental results show that the system has some practical value and research significance.