The high birefringent microstructured optical fiber (MOF) with small defective holes in fiber core and elliptical cladding air holes was designed for polarization-maintaining waveguide transmission for terahertz wave. By selectively filling the terahertz epsilon-near-zero (ENZ) materials in cladding air holes, the dual asymmetry of the geometric structure and the material distribution was introduced, and the degeneracy of two polarized fundamental modes was broken to obtain the high birefringence. The variation of transmission properties such as birefringence, loss and dispersion with structural parameters was studied by finite element method (FEM). In the wide-band range from 0.5 THz~2 THz, the high birefringence greater than 0.01 was obtained. The loss of the x-polarized and y-polarized fundamental mode has the minimum value around 0.8 THz, which are 0.903 dB·cm^-1 and 0.851 dB·cm^-1, respectively. The dispersion characteristics can be effctively adjusted by the defective holes in fiber core, and the y-polarized fundamental mode has the near-zero flattened dispersion characteristics of (0±0.054) ps·THz^-1·cm^-1 in the range of 1 THz~1.8 THz. The transmission characteristics of the designed fiber are insensitive to the refractive index changes of ENZ material. The research conclusions provide theoretical references for the development of terahertz polarization maintaining fiber.