In the stereoscopic display system based on binocular parallax, the improper depth distribution is the main cause of eye fatigue, therefore a novel stereoscopic depth adjustment method is proposed to improve the visual comfort. The proposed method takes the original stereoscopic image as the input. The visual saliency map is derived from original stereoscopic image by applying spectral residual extracting method. Combined with the specific viewing condition, a visual fatigue index model is established to calculate the location of optimum zero disparity plane (ZDP) for specific image in specific viewing condition. The original input stereoscopic image is adjusted according to the optimum ZDP to derive the output stereoscopic image with the minimum visual fatigue index. In order to verify the improvement on visual comfort of the proposed depth adjustment method, the visual perception experiment is designed and implemented. Experimental result indicates that the proposed method significantly improves the stereoscopic viewing comfort.