The remote sensing image should be atmospherically corrected for quantitative application, while removing the adjacency effect is an important step in atmospheric correction. Removing adjacency effect depends on the knowledge of its main impact factors and mechanism. In this study, the main impact factors and mechanism of adjacency effect were investigated, and the adjacency effect of different contrast targets in different conditions was simulated by using MODTRAN model. The results show that the visibility has the first important effect on adjacency effect; the satellite zenith angle has the second one; and the solar altitude angle has the least one. When the visibility is higher, the curve of relatively variable percentage of radiance is more similar to the curve of background radiance. But in the case of lower visibility, this phenomenon is not appeared. When the visibility is higher, the relatively variable percentage of radiance reduces gradually as the wavelength increases. But in the lower visibility, this relation does not appeare. It proves that as the visibility is reduced, its impact on the adjacent effect is more complicated. The results will provide the basis for removing the adjacency effect.