To address the limitations of hardware storage resource and power consumption in infrared-remote-sensing ship detection and the inadequate precision of the output boundary rectangular box form of target detection, a lightweight and pixel-level-output segmentation network TRS-Net (ternary residual segmentation network) is proposed. We apply the encoder-decoder structure of image segmentation to ship detection to obtain the pixel-level output. Further, we binarize the 32-bit floating-point parameters to compress the size of the network model and propose a binary segmentation network (BS-net). Then, to solve the problem of poor detection accuracy caused by BS-Net, we introduce residual connection and propose a binary residual segmentation network (BRS-Net). Furthermore, owing to the sparsity of the neural network, we introduce ternary parameters and propose a ternary segmentation network (TS-Net); therefore, we propose a ternary residual segmentation network (TRS-Net) to further improve the detection effect. Using a long-wave infrared camera independently developed by the laboratory for imaging experiments, we obtain infrared images of ships, make the datasets, and compare and analyze the results of four kinds of networks. The results demonstrate that the detection precision, recall rate, F_1-score, and intersection-over-union of TRS-Net are 88.73%, 83.34%, 85.95%, and 75.36%, respectively. Furthermore, the model size is reduced to one-sixteenth of its original size. Therefore, the proposed TRS-Net has practical engineering value for real-time infrared ship detection.