We theoretically and experimentally demonstrate a cylinder-shaped hole array with a small depth and an appropriate period integrated on a silicon-on-insulator substrate can enhance infrared absorption due to more bending of light and a higher back reflection. The Si metal-semiconductor-metal (MSM) photodiode with an hole array, whose depth is 250 nm, exhibits a 4-fold improved external quantum efficiency (EQE) of 81%, and an ultra-fast impulse response speed of 22 ps enabling a 3 dB bandwidth of up to 23.9 GHz. PbSe film with a thickness of 80 nm is integrated to broaden the response wavelength. A more than 500% EQE enhancement of the Si-based PbSe photodiode with 150-nm-deep photon-trapping holes is achieved at 1550 nm compared to the device without hole structures.