The technique that intense laser pulses is used to anneal the lattice has been established a thermal process since it was discovered in 2074. The thermal model works well for any material that is excited with picosecond or longer-duration laser pulses. For femtosecond and shorter-duration laser pulses, however, the lattice structural changes can be driven directly by electronic excitation. This means that annealing can be completed under melting point. The ultrashort laser pulse annealing is a non-thermal process and it is a new way of annealing. This review focuses on the nature of thermal and non-thermal models. The history, current situation, and the trend of ultrashort laser pulse annealing are also summarized.