Based on the geometrical probability factor of lidar system and the influence of the second mirror projection in the plane of the major mirror of telescope,using Monte Carlo integral method,improves the theoretical equation of geometrical form factor of the off-axis lidar.The optimization of geometrical form factor can be achieved by insert a lens at the focal plane of the telescope with a proper parameter,with the image effect of the lens,ensure the entrance pupil of the echo signal only related to the diameter of the telescope ,which is not affected with the area of the detector.Comparisons analysis between theoretical results and experimental results show that,the proposed method has obvious advantages which the relative error between the calculated value and experimental results reduces 30%.The simulated results prove this enhanced method is effective and simple.