To understand the nanosecond laser derusting process and reveal the derusting mechanism, the surface morphology and roughness of AH32 marine steel after laser derusting under different process parameters was studied. First, a nanosecond-pulse laser was used to clean the rust layer of the sample surface under different process parameters. Then, the surface roughness after cleaning was measured using laser scanning confocal microscopy, the micromorphology was measured by scanning electron microscopy, and an elemental analysis was performed using an energy dispersive spectrometer. Finally, the nanosecond laser derusting mechanism of AH32 marine steel was revealed combined with the experimental results. The experimental results indicated that the cleaned morphology was improved, and that the surface roughness was reduced under stepwise laser cleaning with energy densities of 30.6 and 10.2 J/cm2 at a scanning speed of 3 000 mm/s. Moreover, under the above cleaning conditions, the micromorphology of the matrix exhibited a micromelting state after laser derusting, in which the spot was smooth and uniform and the edge of the spot was distributed with a dendritic mastoid structure. It can be concluded that the use of a stepwise laser cleaning process can yield a better cleaning effect and higher cleaning efficiency. The derusting mechanism mainly includes hole blasting and melting-vaporization.