The derivation of the probability density function (PDF) of differential nonlinear phase noise in diffrential phase-shift keying (DPSK)-balanced direct detection receiver of the high speed optical communication system is achieved by using the Gaussian approximation and Fourier transform methods.Finally,the result is used in computing the bit error rate (BER) considering all the noises influenced the transmission performance of the DPSK-balanced receiver system.The results have been verified by simulations through VPI software.