Aiming at the problem of small target recognition and segmentation, a network model based on bilateral fusion (BFNet) is proposed, which has a dual-branch structure. One has a narrow channel and a shallower structural layer, which focuses on the connections between adjacent pixels. The other introduces two modules, such as receptive field block (RFB) and dense fusion block (DFB), which have wider channels and deeper structural layers and can obtain high-level semantic context information. It is then represented by a guide aggregation layer that fuses the features of the two branches. Three open medical segmentation datasets are used to evaluate the performance of the proposed algorithm. The experimental results show that the proposed algorithm is superior to existing medical image segmentation algorithms in the segmentation task of polyps and skin lesions. Especially, in the automatic polyp detection Kvasir-SEG dataset, the average Dice and average cross ratio of the proposed algorithm reached 92.3% and 86.2% respectively, which are both higher than the existing algorithms.