Point target detection in Infrared Search and Track (IRST) is a challenging task. due to less information. Traditional methods based on hand-crafted features are hard to finish detection intelligently. A novel deep spatial-temporal convolution neural network is proposed to suppress background and detect point targets. The proposed method is realized based on fully convolution network. So input of arbitrary size can be put into the network and correspondingly-sized output can be obtained. In order to meet the requirement of real time for practical application, the factorized technique is adopted. 3D convolution is decomposed into 2D convolution and 1D convolution, and it leads to significantly less computation. Multi-weighted loss function is designed according to the relation between prediction error and detection performance for point target. Number-balance weight and intensity-balance weight are introduced to deal with the imbalanced sample distribution and imbalanced error distribution. The experimental results show that the proposed method can effectively suppress background clutters, and detect point targets with less runtime.