A physical model for simulating overlay metrology employing diffraction based overlay (DBO) principles is built. It can help to optimize the metrology wavelength selection in DBO. Simulation result of DBO metrology with a model based on the finite-difference time-domain (FDTD) method is presented. A common case (bottom mark asymmetry) in which error signals are always induced in DBO measurement due to the process imperfection were discussed. The overlay sensitivity of the DBO measurement across the visible illumination spectrum has been performed and compared. After adjusting the model parameters compatible with the actual measurement conditions, the metrology wavelengths which provide the accuracy and robustness of DBO measurement can be optimized.