Based on coupled-mode theory, the mode transition of the high-order cladding modes in a coated long-period fiber grating (LPFG) has been studied firstly; the mode transition region and non-mode transition region of high-order cladding modes are divided. The response characteristic of cladding mode effective index with increasing the overlay thickness is analyzed; the shift of resonant wavelength in the mode transition region will be larger than that in the non-mode transition region. Further, the changes of the resonant wavelength of some high-order cladding modes with grating period are investigated when the cladding radius are different, the shift between two resonant wavelengths of dual peak in the mode transition region is bigger than that in non-mode transition region when the cladding radius are uniform. And the shift between two resonant wavelengths of dual peak will be increased by the decrease of the cladding radius in both mode transition and non-mode transition regions. Finally, the response characteristics of film refractive index of coated LPFG are investigated for a high-order cladding mode while the cladding radius are different and the overlay thickness is located in mode transition region and non-transition mode region, then the optimized design scheme is come up with. The higher sensitivity dual-peak sensor of coated LPFG than the traditional dual-peak sensor will be obtained when the overlay thickness and refractive index is located in the mode transition region and the grating period close to the phase matching turning points. Further, the resolution power of coated LPFG sensor will further be improved by the appropriate reducing of the cladding radius.