A kind of terahertz metamaterial with a near-zero magnetic permeability (μ) which uses a metal-dielectric composite structure as the basic unit cell is proposed. Under the condition that the terahertz wave is incident perpendicularly, the optimal value of the real part of μ is 0 near the resonance frequency of 1.44 THz with a bandwidth (|μ|<0.05) of about 0.2 THz when the electric field is transverse-magnetic polarized. However, when the electric field is transverse-electric polarized, the optimal value of the real part of μ is 0 near the resonance frequency of 0.978 THz with a bandwidth (|μ|<0.05) of about 0.1 THz. Under the fixed geometric parameters, the μ-near-zero effect can be achieved in different frequency bands via the adjustment of polarization direction. The influences of the thickness of polyimide dielectric layer, metal layer number, and incident angle on the μ-near-zero effect are analyzed, and the tolerance error range of this structure is discussed.