Optical scattering characteristics of high altitude balloon in near space are researched based on the fundamental radiation theory in applied optics. Facet of balloon in near space is created according to the general idea of coordination transformation and surface mesh-creation. A bidirectional scattering distribution function (BSDF) model is deduced by summing transparent object all scattering components, including specular reflective, near-specular reflective, diffuse reflective and random diffuse reflective components. A mathematical model of scattering characteristics of balloon in near space is built. An atmospheric modeling tool, MODTRAN, is used to model background radiance in the range of 3~5 μm and 8~14 μm and radiance of balloon in the range of 0.24~2.4 μm . The results show that the high altitude balloon radiance is 2.28 × 10-3 W ∕(cm2·μm·sr) , whose calculation error is 10.6% in use of BSDF model, and relative to bidirectional reflection distribution function (BRDF) model, the accuracy has an improvement of 2%. The modeling method of scattering characteristics of balloon in near space agrees with the modeling results, which proves the correctness of mathematical model.