Position sensitive detector (PSD) has a great advantage in terms of visual pose measurement due to less affect by light and the simple data processing. However, due to the non-linearity of the PSD sensor itself, the distortion of the PSD sensor is complex and the calibration of the PSD sensor becomes the key to its application. Aiming at the PSD vision measurement system composed of PSD camera and infrared light emitting diode (LED) light source, we propose a PSD camera calibration method. The method uses polynomial model to characterize the PSD camera distortion, combined with the traditional pinhole imaging model to form the PSD camera imaging model. Non-linear optimization is used to get calibration camera model parameters. A dedicated infrared LED calibration board is designed to achieve PSD vision measurement system calibration. Experimental results show that this method has high calibration accuracy for PSD camera. With this calibration result, we can obtain high measurement accuracy of position and orientation and short running time, which can meet the demand of high precision optical measurement.