An empirical method for synthetic aperture radar (SAR) sea significant wave height (SWH) inversion based on the extreme learning machine (ELM) model is proposed in this study. Spatial-temporal-matched ENVISAT ASAR image and ECMWF reanalysis sea SWH dataset are collected and analyzed to establish the empirical method. Two cases of mass- and less-matched datasets are used to investigate the capability of the ELM model to establish the empirical relationship from the SAR image parameters to wave SWH parameters. In addition, the CWAVE method is compared with established empirical method as a reference. Results show that the training precision of empirical method is 0.87 in the case of mass-matched dataset, which is slightly worse than that of the CWAVE algorithm (0.91). However, in terms of the method training efficiency, the empirical method (0.022 s) behaves better than the CWAVE algorithm (0.514 s). Moreover, in the case of less-matched dataset, the training accuracy of empirical method is 0.59 and its training efficiency is 0.008 s, which is much better than the CWAVE algorithm (-0.38 and 0.318 s), revealing that the ELM-based empirical method can achieve high retrieval precision of SAR wave SWH results under less-matched dataset.