The coupling effect in three dimensional measurements using optical microscopy refers to the phenomenon that the accuracy of the measurement height of a groove or step sample that suffers a principle error caused by the influence of the relatively small transverse period of the sample. Based on the convolution irrelevance principle and the limited energy lost principle, two models depicting the coupling effects in measurements of the thin step samples and the deep groove samples were established respectively, thus the coupling relationship between the geometric parameters of the sample and the measurement capability of the optical instrument were revealed. Compared with the existing W/3 reading rule, this criterion can more objectively reflect the different influences of samples with different structures on the actual measurement capability of the optical instrument, and indicate the assessment portions for height measurements without accuracy loss, also is able to predict the generation of principle error in height assessment if the structure is too tight or deep, thus provides a new reading and evaluation criterion for the three dimensional measurement of a groove or step sample.