A novel single-polarization single-mode (SPSM) rhombic-hole terahertz (THz) photonic crystal fiber (PCF) is proposed. The influence on the characteristics of SPSM is studied by changing the number, size and location of the defects in fiber. A full-vector finite element method is used for modeling, and the simulation results are as follows. In the case of two defects in PCF, with the increase of defect size, SPSM operation range moves towards high-frequency, and its width is increased slightly. In the case of four defects in PCF, the characteristics of SPSM is affected by the defects near the core region easily. Based on this conclusion, four same defects are introduced around the core region in fiber. Compared with the case of two defects, SPSM operation range (1.07 THz to 1.36 THz) moves towards high-frequency and its width (0.29 THz) increases significantly for four defects case, which is 1.32 times than the case of two defects. So the fibers have wide applications in some systems that have strict requirements about the polarization states, and have practical significance.