A holographic wave-front sensor (HWFS) is a modal wave-front sensor, which utilizes a multi-mode composite holographic element and a position sensitive photodetector to extract the amplitudes of specific Zernike modes from the incident light. However, disturbing light from high order diffraction of the multi-mode composite holographic element is inevitable. The disturbing light disperses the power of the incident light and decreases the signal-to-noise ratio of the sensor. Here, an intensity modulation method is introduced to eliminate the interference light generated by the diffraction of binary phase holographic elements. The proposed method can be used for the elimination of interference light in the holographic mode wavefront sensor, and it is suitable to other applications based on binary phase holographic elements, such as holographic imaging, phase-stepping interferometers, and beam deflectors.