In view of the problems that the target contour of the fused image is fuzzy and its details are not highlighted by the use of traditional fusion algorithms, an infrared and visible image fusion algorithm based on the shearlet frame and neighborhood structure features is proposed. The shearlet transform is used to decompose the source images to get the subbands coefficients of high frequency and low frequency with the same size as the original images. Then, in order to prevent the edge of the fusion image from blurring after fusion, a fusion rule based on geometrical distance combined with energy distance is adopted in low frequency subband coefficients. Moreover, a fusion strategy based on gray difference and gradient distance weighting is used to fuse high frequency subband coefficients for keeping the details of the images better. Finally, the fusion image is obtained by shearlet inverse transformation. Results show that the proposed algorithm can effectively extract the target infrared information and keep the visible image information. On the basis of retaining the image profile information,the proposed algorithm can highlight the target information, and improve the image fusion effect effectively.