Two mainstream approaches for solving inverse sample reconstruction problems in programmable illumination computational microscopy rely on either deep models or physical models. Solutions based on physical models possess strong generalization capabilities while struggling with global optimization of inverse problems due to a lack of sufficient physical constraints. In contrast, deep-learning methods have strong problem-solving abilities, but their generalization ability is often questioned because of the unclear physical principles. In addition, conventional deep models are difficult to apply to some specific scenes because of the difficulty in acquiring high-quality training data and their limited capacity to generalize across different scenarios. To combine the advantages of deep models and physical models together, we propose a hybrid framework consisting of three subneural networks (two deep-learning networks and one physics-based network). We first obtain a result with rich semantic information through a light deep-learning neural network and then use it as the initial value of the physical network to make its output comply with physical process constraints. These two results are then used as the input of a fusion deep-learning neural work that utilizes the paired features between the reconstruction results of two different models to further enhance imaging quality. The proposed hybrid framework integrates the advantages of both deep models and physical models and can quickly solve the computational reconstruction inverse problem in programmable illumination computational microscopy and achieve better results. We verified the feasibility and effectiveness of the proposed hybrid framework with theoretical analysis and actual experiments on resolution targets and biological samples.