Based on the Kirchhoff-Fraunhofer diffraction integral, a theoretical study, with reference to experimental results, of transverse self-phase modulation effects in the transmission of a laser beam through a homeotropic C60-doped nematic liquid crystal (5CB) cell was presented. Two different type of concentric diffraction rings were observed in the far field. The simulation results show that, the diffraction rings were determined by the two factors of C1 and C2. When the sign of C1 is identical to that of the C2, the pattern in the far field is a series of thin, narrow diffraction rings with a central bright spot. When the sign of C1 is opposite to that of the C2, the thick diffraction ring pattern with central dark spot and the large distribution range will emerge in the far-field plane. The phenomenon is attributed to the spatial self-phase modulation caused by the refractive index change of the medium when the Gaussian beam passes through the sample. The results show that the theoretical analysis is consistent with the observed experimental phenomena, which is significant in the applications of practical nonlinear optical limiters for the eye or sensor protection.