If the geometrical structure of a rib waveguide with large cross section meets the conditions of single mode, the waveguide device does not necessarily work in the best status. The interference of leaky modes often cannot be ignored. The scalar finite difference beam propagation method is used to calculate the attenuation degree of leaky modes supported by the single-mode rib waveguides with different geometric dimensions. When the attenuation is the largest, the geometrical structure of rib waveguide is the most optimal. The single-mode condition is still satisfied after the optimization, but the attenuation of leaky modes is very large and the interference to the main mode is ignored. For a typical single-mode rib waveguide, whose width of rib is 2 μm and ratio of rib height to total height is fixed at 0.45, when the total height as variable parameter ranges from 5.4 μm to 1.3 μm, the optimized total height is 2.25 μm after the calculation of attenuation of leaky modes with different geometric dimensionings.