According to the requirements of panorama assistance driving systems, a non-spherical fish eye lens is designed. The design parameters and optical materials of the lens are determined, and the initial structure of the lens is built. Through the aberration correction and optimization design of the lens, the design results are given. The designed non-spherical fish eye lens, distortion correction algorithm and overall hardware are implemented and verified to integrated with the panoramic display driver assistance system. The images captured by multiple lenses are synthesized and applied, and the experimental results show that the system has good application.