The ultra-low thermal conductivity of roughened silicon nanowires (SiNWs) can not be explained by the classical phonon–surface scattering mechanism. Although there have been several efforts at developing theories of phonon–surface scattering to interpret it, but the underlying reason is still debatable. We consider that the bond order loss and correlative bond hardening on the surface of roughened SiNWs will deeply influence the thermal transport because of their ultra-high surface-to-volume ratio. By combining this mechanism with the phonon Boltzmann transport equation, we explicate that the suppression of high-frequency phonons results in the obvious reduction of thermal conductivity of roughened SiNWs. Moreover, we verify that the roughness amplitude has more remarkable influence on thermal conductivity of SiNWs than the roughness correlation length, and the surface-to-volume ratio is a nearly universal gauge for thermal conductivity of roughened SiNWs.