Nonlinear silicon photonics has shown an ability to generate, manipulate, and detect optical signals on an ultracompact chip at a potential low cost. There are still barriers hindering its development due to essential material limitations. In this review, hybrid structures with some specific materials developed for nonlinear silicon photonics are discussed. The combination of silicon and the nonlinear materials takes advantage of both materials, which shows great potential to improve the performance and expand the applications for nonlinear silicon photonics.