Rhythmic behaviors, i.e. temporally periodic oscillations in a system, can be ubiquitously found in nature. Interactions among various rhythms can lead to self-organized behaviors and synchronizations. This mechanism is also responsible for many phenomena such as nonlinear waves, spatiotemporal patterns, and collective behaviors in populations emerging in complex systems. Mathematically different oscillations are described by limit-cycle oscillators (pacemakers) with different intrinsic frequencies, and the synchrony of these units can be described by the dynamics of coupled oscillators. Studies of microscopic dynamics reveal that the emergence of synchronization manifests itself as the dimension reduction of phase space, indicating that synchrony can be considered as no-equilibrium phase transition and can be described in terms of order parameters. The emergence of order parameters can be theoretically explored based on the synergetic theory, central manifold theorem and statistical physics. In this paper, we discuss the order-parameter theory of synchronization in terms of statistical physics and set up the dynamical equations of order parameters. We also apply this theory to studying the nonlinear dynamics and bifurcation of order parameters in several typical coupled oscillator systems.