The imaging circle of panoramic fisheye optical system is usually a complete circle, which is smaller than the vertical size of the sensor. Compared with rectangular sensor, the area of circular imaging circle is smaller and the utilization rate of effective pixels of sensor is low. Combined with the characteristic that the imaging circle of panoramic fisheye optical system is smaller than that of sensor, a panoramic fisheye optical system based on freeform surface design was introduced, which could realize elliptical imaging area and greatly improve the utilization rate of effective pixels of sensor. The freeform surface model was built by using the optical design software, which was composed of glass lens and plastic lens. The imaging area of the asymmetric panoramic fisheye optical system was designed to be ellipse by eliminating aberration and controlling the focus shift under different temperature fields. The X direction of the lens image height was close to the horizontal size of the sensor, and the Y direction of the lens image height was close to the vertical size of the sensor. Without considering the manufacturing tolerance, according to the circle and ellipse area formula, for sensor of 4:3, the effective utilization rate of sensor pixels in the circular imaging area was about 58.9%, and that in the elliptical imaging area was about 78.5%. Finally, the simulation and calculation results show that the utilization rate of effective pixels of the sensor with elliptical imaging optical design is about 15% higher than that with circular imaging optical design.