The diffraction efficiency of the traditional single-layer diffractive optical elements (SLDOEs) decreases drastically when the wavelength deviates from the center wavelength, causing image blurring when the elements are used in a wide-waveband infrared system. In this study, we analyze the influence of the field of view on the point spread function (PSF) model of the system containing diffractive optical elements. In addition, we propose an optical-digital joint design method for reducing the influence of the field of view. In this method, the PSF is constant with respect to the field of view by assigning optimization weights to different diffractive orders. Further, the center wavelength of the SLDOE is designed in the medium waveband. The PSF model is used for reconstructing the blurred image in the long waveband to achieve dual-waveband imaging. A dual-waveband infrared refractive-diffractive system is designed using this method. The MTF with respect to the medium waveband at 17 lp/mm is 0.7. The reconstructed image in the long waveband is subjectively clear, and several image quality assessment function values are improved. Results show that the proposed design method enables the application of SLDOEs in dual-waveband infrared systems.