The nondiagonalization method is proposed for analyzing competition and transformation of the transverse modes in hollow waveguide resonators. From calculating transformation matrix of propagation for the EH1m modes, we find that the energy of transverse modes of EH1m can be transformed from each other, and the engery transformation is in general unequal. It is shown that the lowest order transverse mode (EH11) is favoured when the cavity parameters are correctly chosen.