Recently, deep learning has been used to establish the nonlinear and nonintuitive mapping between physical structures and electromagnetic responses of meta-atoms for higher computational efficiency. However, to obtain sufficiently accurate predictions, the conventional deep-learning-based method consumes excessive time to collect the data set, thus hindering its wide application in this interdisciplinary field. We introduce a spectral transfer-learning-based metasurface design method to achieve excellent performance on a small data set with only 1000 samples in the target waveband by utilizing open-source data from another spectral range. We demonstrate three transfer strategies and experimentally quantify their performance, among which the “frozen-none” robustly improves the prediction accuracy by ∼26 % compared to direct learning. We propose to use a complex-valued deep neural network during the training process to further improve the spectral predicting precision by ∼30 % compared to its real-valued counterparts. We design several typical teraherz metadevices by employing a hybrid inverse model consolidating this trained target network and a global optimization algorithm. The simulated results successfully validate the capability of our approach. Our work provides a universal methodology for efficient and accurate metasurface design in arbitrary wavebands, which will pave the way toward the automated and mass production of metasurfaces.