Underwater images often suffer from low contrast, color distortion, and poor visibility. To solve these problems, herein a novel underwater image restoration method based on scene depth estimation and background segmentation is proposed. First, the scene depth is estimated using multiple oblique gradient operators and attenuation difference among color channels. Then, according to the image gradient and color difference information, the degraded underwater image is divided into the foreground region and the background region. Accordingly, the background light (BL) is estimated in the background region and transmission maps are obtained using the estimated scene depth map. Subsequently, the scene radiance of the foreground region is recovered based on the underwater image formation model, and the background region is enhanced by performing histogram stretching in the HSV color space. Finally, the foreground and background are fused using a weight map of the transition region to obtain the final restoration result. Experimental results show that the proposed method can estimate the background light and transmittance with significantly greater accuracy, and achieves satisfactory contrast enhancement, color correction, and sharpness improvement. Compared with several classical methods, the proposed method affords 15% better performance on average in terms of the following four image quality evaluation metrics: UIQM, UCIQE, FDUM, and FADE.