For the holey fiber with diamond lattice, the influence of the special holes near the core, the bottom holes and the pitch of diagonal holes on the wavelength sensitivity of polarization beat length is analyzed. The preliminary optimal cladding structure parameters are obtained. Then two new asymmetric structures are introduced through the lateral deformation. The holey fiber original non-linear variation of birefringence with wavelength is further suppressed. The wavelength sensitivity of the polarization beat length is reduced while a larger error tolerance of the cladding structure geometrical parameters is obtained. The results show that the median value of polarization beat length is about 132.5 mm and the relative variation of polarization beat length is less than ±4% in wavelength range of 1.2~1.7 μm while the error tolerance of optimized geometric parameters is greater than 0.1 μm. The more feasible process structure parameters are provided for drawing the achromatic holey fiber wave plates with broadband.