Imaging through scattering media is valuable for many areas, such as biomedicine and communication. Recent progress enabled by deep learning (DL) has shown superiority especially in the model generalization. However, there is a lack of research to physically reveal the origin or define the boundary for such model scalability, which is important for utilizing DL approaches for scalable imaging despite scattering with high confidence. In this paper, we find the amount of the ballistic light component in the output field is the prerequisite for endowing a DL model with generalization capability by using a “one-to-all” training strategy, which offers a physical meaning invariance among the multisource data. The findings are supported by both experimental and simulated tests in which the roles of scattered and ballistic components are revealed in contributing to the origin and physical boundary of the model scalability. Experimentally, the generalization performance of the network is enhanced by increasing the portion of ballistic photons in detection. The mechanism understanding and practical guidance by our research are beneficial for developing DL methods for descattering with high adaptivity.