Photodetectors play a key role in many applications, such as remote sensing, night vision, reconnaissance, medical imaging, thermal imaging, and chemical detection. With the increasing complexity of photoelectric detection tasks, photodetectors working in different bands are gradually integrated into broad spectral detection for the same scene. Limited by the volume and task module of the integrated system, conventional broad spectral detection tasks often require multiple detectors working in different bands to perform together, which greatly increases the complexity of detection system. Therefore, photodetector enabling to response ultra-broadband radiation (UV-vis-IR-THz) has gradually become a subject of great interest in recent years. However, there have been no reports on the review of ultra-broadband photodetectors so far. Hence, this review systematically summarizes the research progresses of ultra-broadband photodetectors in the past ten years. The factors affecting the response performance of photodetectors and the main types of common photodetectors are described first, and then the research progress, development status and challenges are reviewed and suggestions about the future research directions of ultra-broadband photodetectors are also provided.