A novel approach to design the subwavelength metal rectangular gratings used as the polarizing beam splitters was presented. The method was based on effective medium theory (EMT) and thin film optics. The polarizing diffraction performance of subwavelength grating was analyzed by the method of rigorous couple-wave analysis (RCWA) for the wavelength of 650 nm. The numerical calculation was used to analyze the effects of the 0-order diffraction efficiency caused by the grating period,the groove depth,the incident angle and the incident wavelength. The results show that subwavelength metal rectangular grating have the characteristics of metal films with high reflectivity for TE polarization and the characteristics of the dielectric films with high transmission for TM polarization. The optimal design agrees excellently well with the results simulated by RCWA method.