Since the current image quality assessment methods are generally based on hand-crafted features, it is difficult to automatically and effectively extract image features that conform to the human visual system. Inspired by human visual characteristics, a new method of full-reference image quality assessment was proposed by this paper which was based on convolutional neural network (DeepFR). According to this method, the DeepFR model of convolutional neural network was designed which was based on the understanding of the dataset by itself using the human visual system to weight the sensitivity of the gradient, and the visual gradient perception map was extracted that was consistent with human visual characteristics. The experimental results show that the DeepFR model is superior to the current full-reference image quality assessment methods, its prediction score and subjective quality evaluation have good accuracy and consistency.