In order to solve the problems of binocular camera′s optical axis alignment error, low image matching accuracy, and insufficient computing speed in passive binocular ranging systems, a new self-calibrated binocular ranging system based on hardware speeded up robust features (SURF) algorithm is proposed. The scheme uses position sensitive detectors and servo motors as the self-calibrating platform of the binocular cameras, and Zynq SoC is utilized to achieve the hardware acceleration of the SURF algorithm and feature matching of two images. Finally accurate and effective passive ranging is realized. The results show that the position sensitive detector can effectively calibrate the binocular cameras to the same plane position, and the hardware SURF algorithm can match images with high accuracy and speed. The scheme meets the demand of passive ranging systems.