It is highly desirable to flexibly and actively manipulate the dephasing time of a plasmon in many potential applications; however, this remains a challenge. In this work, by using femtosecond time-resolved photoemission electron microscopy, we experimentally demonstrated that the Fano resonance mode in the asymmetric nanorod dimer can greatly extend the dephasing time of a femtosecond plasmon, whereas the non-Fano resonance results in a smaller dephasing time due to the large radiative damping, and flexible manipulation of the dephasing time can be realized by adjusting one of the nanorods in the Fano asymmetric dimer. Interestingly, it was found that plasmon resonance wavelengths both appeared red-shifted as the length of the upper or lower nanorods increased individually, but the dephasing time varied. Furthermore, it also indicated that the dephasing time can be prolonged with a smaller ascending rate by increasing the length of both the nanorods simultaneously while keeping the dimer asymmetry. Meanwhile, the roles of radiative and nonradiative damping in dephasing time are unveiled in the process of nanorod length variation. These results are well supported by numerical simulations and calculations.