A model of electron interacting with nested X-ray focusing optics was designed firstly. Then the process of electrons interacting with coating material of X-ray focusing optics and the quantum efficiency of X-ray source by using a Monte Carlo N Particle transport code was analysized. Simulation results accord well with the analyze results. Finally, a silicon drift detector, which has an energy resolution of 125 eV was used to analysis the energy spectrum in the focal point. In the detecting part, X-ray photons are composed of two parts, signal photons generated by the X-ray source and noisy X-ray photons generated from nested X-ray focusing optics′ coating material. According to different conditions of X-ray source and electron gun voltage, the number of signal photon, noisy photon and signal to noise ratio of the X-ray communication demonstration system then can be calculated. Experiment and calculate results show that nested X-ray focusing optics can effectively filter spatial particles, which will optimize the signal to noise ratio of X-ray communication system. When the number of incident electrons is about 1×108 counts per second with an energy distribution from 1~20 keV, signal to noise ratio of the X-ray communication demonstration system can reach at least 15.1 dB. These results will provide foundations for optimizing the core parameters of X-ray communication system in the future.