Metasurfaces composed of spatially arranged ultrathin subwavelength elements are promising photonic devices for manipulating optical wavefronts, with potential applications in holography, metalens, and multiplexing communications. Finding microstructures that meet light modulation requirements is always a challenge in designing metasurfaces, where parameter sweep, gradient-based inverse design, and topology optimization are the most commonly used design methods in which the massive electromagnetic iterations require the design computational cost and are sometimes prohibitive. Herein, we propose a fast inverse design method that combines a physics-based neural network surrogate model (NNSM) with an optimization algorithm. The NNSM, which can generate an accurate electromagnetic response from the geometric topologies of the meta-atoms, is constructed for electromagnetic iterations, and the optimization algorithm is used to search for the on-demand meta-atoms from the phase library established by the NNSM to realize an inverse design. This method addresses two important problems in metasurface design: fast and accurate electromagnetic wave phase prediction and inverse design through a single phase-shift value. As a proof-of-concept, we designed an orbital angular momentum (de)multiplexer based on a phase-type metasurface, and 200 Gbit/s quadrature-phase shift-keying signals were successfully transmitted with a bit error rate approaching $1.67\times {10}^{-6}$. Because the design is mainly based on an optimization algorithm, it can address the “one-to-many” inverse problem in other micro/nano devices such as integrated photonic circuits, waveguides, and nano-antennas.