The bit error rate (BER) of optical code-division multiple-access (OCDMA) system is an important index of evaluating system performance because it can reflect the premise of considering dispersion and nonlinear effect broadening in single-mode fibers. Based on the definition of BER in ideal condition,the influences of third-order dispersion (TOD) and third-order nonlinear on the BER are introduced and the BER of OCDMA system with optical hard-limiter (OHL) and 2D OCDMA system is analysed. The results show that BER will increase under the action of TOD and third-order nonlinear which has lower effect on system performance than TOD,and at the same time the performance of transmission system with OHL and using two-dimensional code will be improved under the same fiber condition.