Traditional laser linewidth characterization is usually done by introducing a self-heterodyne technique. This technique is an optical fiber delay based on laser beat note, which generates a Lorentzian spectrum related to the linewidth of a laser. In order to obtain the complete characteristics of a laser linewidth and frequency noise spectrum in its frequency domain, a new method based on β algorithm is proposed. The basic principles of the β algorithm have been analyzed and explained at first. The relationship between frequency noise and laser linewidth in different frequency ranges are analyzed based on the Wiener-Khintchine theorem. When the cut-off frequency tends to be zero or infinity, the laser line shape evolves from Gaussian type to Lorentzian type. Meanwhile, the cut-off frequency of the conversion laser line shape has been deduced, which is represented by the frequency noise function, namely β separation line. Secondly, the frequency noise spectral density has been measured with OE4000 test system. The frequency noise and the line shape of diode lasers are numerically simulated. In the low frequency region, where the noise level is much larger than its frequency, it produces a slower frequency modulation than that in the high frequency region. The linewidth of the laser is an integration of the frequency noise in the Gaussian line shape region. The error in linewidth calculation is smaller over the entire cut-off frequency range. Finally, a set of measured frequency noise power spectral density of a RIO’s laser was used to calculate the linewidth using the β algorithm. When the frequency noise spectral density is greater than the β separation line, the laser appears as a Gaussian line shape and the linewidth decrease with frequency banwidth; on the contrary, the laser shows a Lorentzian line shape, and the linewidth is fixed. Meanwhile, a delayed self-heterodyne measurement system with delay fiber of 50 km and frequency shift of 60 MHz is constructed. The measured linewidth of a RIO’s laser working under 110 mA inject current is about 1.8 kHz, which is consistent with the calculated result of the β algorithm, of which the frequency bandwidth is ~2.8 kHz. In conclusion, β algorithm is able to characterize the linewidth of any type of narrow linewidth laser, which is of great significance to the research on narrow linewidth lasers.