An information extraction method based on a panoramic stereo imaging system with double coaxial panoramic annular lens (PAL) units is presented. Its imaging circles are concentric so that depth information of 360° horizontal field-of-view (FOV) object can be extracted in real time. Based on imaging principles, the system is calibrated with a model that can accommodate the characteristics of a non-single viewpoint system, and the calibration results are verified. The scale-invariant feature transform (SIFT) algorithm is applied to the corresponding point matching, and it makes full use of collinear constraint of the system to improve the matching speed and accuracy. Effective depth information is extracted from the captured image with applying triangulation principle. Depth information extraction experiments are performed, and the results and error analysis are given. This system can restore depth information effectively, and its error rate of 3-m distance range is within ±5.2%, which proves the feasibility of this system.