Monolayer hexagonal BN (hBN) material served as the nucleation layer in AlN epilayer by metal-organic chemical vapor deposition. Through this method both the stress and the cracks in AlN films were greatly suppressed. The monolayer hBN material was treated with artificial surface chemical modification to increase the defects in hBN and the nucleation sites for the subsequent AlN growth. We analyzed the quality of AlN materials with/without the hBN nucleation layer. The results show the hBN layer can help decrease the cracks, air voids and stress in AlN films. Meanwhile, the impacts of different V/III ratios on the surface morphology, the crystal quality and the stress in AlN films were studied. An optimized V/III-ratio range was obtained. Under the optimized growth condition, the stress in the AlN layer with the hBN nucleation layer can be eliminated completely, and the crystal quality of the AlN films with a hBN nucleation layer was comparable with that of AlN directly grown on sapphire.