At present, the spectrum evolution of the beat signal can be used to measure the fiber nonlinear coefficient. By theoretical simulations, the effect of the stability of the output power, the frequency and the linewidth of the laser on the measurement of the nonlinear coefficient were analyzed. The simulations agree well with the experiments. The propagation properties of a kind of photonic crystal fiber (PCF) are analyzed with the compact supercell method. Taking self-phase modulation (SPM), fiber loss and dispersion into account, the transmission process and the evolution spectra of the beat signal propagating along the PCF are analyzed by using the split-step Fourier method. The results show that the stability of the laser output power has little effect on the measurement of the nonlinear coefficient, but the linewidth and the stability of the central wavelength have more effects on it. Proper parameters should be established to get a more accurate measurement result, in order of the nonlinear coefficient.