We use Monte Carlo simulation and optical density algorithm to evaluate the detection performance of line-scan imaging system for internal defects of tested samples in this paper. First, a fine division for the irregular tissue boundary of the internal defects is achieved using a three-dimensional voxel segmentation method, as it is difficult to accurately simulate the optical transmission of complex tissues by the traditional Monte Carlo method. Then, the effects of the instrument parameters on the penetration depth of photons in the tissue, the detection depth of the detector, and the diffuse reflectance of surfaces are analyzed, and the optimal parameter configuration is determined. Finally, the optical density algorithm is used to evaluate the detection performance of the system for defects with different sizes and depths. The simulation results show that the line-scan imaging detection system can achieve a good balance between the photon detection depth and the surface reflectivity, under a light source with an incident angle of 15° and a distance of 1 mm between the light source and detector. For large (a=2 mm, b=3 mm, c=1 mm), medium (a=2 mm, b=2 mm, c=1 mm), and small (a=2 mm, b=1.5 mm, c=1 mm) ellipsoid defects, the defect depth detection limits of the system are 3.5 mm, 3 mm, and 2.7 mm, respectively. Hence, the study provides a theoretical basis for parameter optimization and performance evaluation of the line-scan imaging system for detecting the internal defects in agricultural products such as fruits.