The properties of leaky modes of the fluorescence and the amplified spontaneous emission (ASE) in the dye film with quasi-waveguide structure are theoretically and experimentally investigated. Theoretically, considering the propagation loss and dye absorption loss, the physical mechanism of leaky-modes characteristics of the fluorescence and ASE are analyzed by calculating the field intensity distributions of the leaky-modes in the dye film with quasi-waveguide structure. Experimentally, the dye films are spin coated on the prism substrate, and the spectra of fluorescence and ASE of leaky modes of the dye films with quasi-waveguide structure are measured by the prism coupling method at different wavelengths and different output angles. The results show the consistence of theoretic analysis and experimental measurement. In addition, a beam analyzer is used to obtain the intensity distribution of fluorescence and ASE of leaky modes to understand the ASE exciting characteristic in the dye film with quasi-waveguide structure. It shows the redshifts of the fluorescence peaks and the ASE peak as well as the exciting leaky modes come from the interaction of the interface reflectivity, the propagation loss and the dye absorption loss, while the interface reflectivity is the dominate factor for exciting the leaky-modes, and the dye self-absorption is the dominate factor for the redshift of fluorescence peaks and the ASE peaks.