To meet the assembling requirements of the optical system of a certain Helmet Mounted Display(HMD), features of the lens in manufacturing, testing and assembly were investigated. And finally, an effective way that could solve problems in assembly of off-axis optical system was proposed. With modern production technologies, an adjusting device with five degrees of freedom was designed. After error analysis and active compensation to parallelism error of guide rail, the adjusting precision of angle for the device is 3.15″, while the displacement adjusting accuracy is about 1μm. Furthermore, surface modeling was brought into the design of lens tube, which adopted a thin shell and was divided in two parts, to get compact structure. After comparison, aluminum alloy was selected as the material for lens tube, and some essential analyses were given to the single tube unit. The results showed that the total mass of the tube unit is 195 g, while the maximum stress acting on it is only 1.5 MPa, with the deformation of 0.3 μm. It turned out that the solution satisfied the requirements of design.