The sea environment is complex, and the lidar installed on the unmanned surface vehicle will generate various useless clutters in the process of detecting the target near our vehicle. And the useless clutters will affect the detection and tracking process and threaten the safety of the unmanned surface vehicle in the process of autonomous voyage. Therefore, it is necessary to effectively filter out all kinds of useless clutters generated by lidar on the sea. In this paper, the data association between the lidar and panorama camera is carried out to achieve the pixel-level registration of 3D point clouds and panoramic images in 2D images. According to the association results of the two kinds of data and the detection of target ships on the panoramic image, the three types of point cloud clutters defined in this paper are filtered.