Aiming at the micro-video popularity prediction, we propose a micro-video popularity prediction model with a bidirectional deep encoding network. The model considers both multi-modal fusion and unimodal supervision modeling, and integrates them into a bidirectional deep encoding network. The multi-modal fusion module uses modal relevance to solve problems such as data missing and dimensional differences among original features to obtain a more comprehensive feature representation. The unimodal supervision module uses modal differences to supervise multi-modal feature fusion. Via joint training of multi-modal fusion and unimodal supervision tasks, the consistency and difference of multi-modal information are fully learned to improve the generalization ability of the algorithm. The experiments on the public NUS dataset have proved the effectiveness and superiority of our proposed algorithm.