The nano groove can highly improve the transmittance of coaxial nanoring aperture due to the excitation of surface plasmon polariton (SPP). The total angular momentum carried by incident beam is reserved in the whole process and transferred to the SPP, thus the vortex SPP carrying orbital angular momentum is generated. The enhanced transmittance of nano aperture by vortex SPP has a wide range of applications, but its physical mechanism has been ignored for a long time. Here we study the process of the enhanced transmittance of the coaxial nanoring aperture and provide a model to describe the process of photon transmission. When the incident light irradiates on the coaxial nanoring aperture and nano groove, the vortex SPP induced by the groove propagates to coaxial nanoring aperture. Some of the photons in the SPP are coupled into the coaxial nanoring aperture and some are reflected back. The reflected photons travel back and forth multiple times between the coaxial nanoring aperture and nano groove. The vortex SPP interacts with the incident beam at the round of coaxial nanoring aperture, which determines the intensity at the round of the coaxial nano aperture, and thus affecting the transmittance. We systematically study the influence of optical angular momentum and the radius of the nano groove on the transmittance of coaxial nanoring aperture by using theoretical analysis and numerical simulations. The results show that the optical angular momentum and radius of the nano groove both affect the radial propagation phase of vortex SPP from nano groove to coaxial nanoring aperture, hence affecting the intensity of the electric field at the round of coaxial nanoring aperture and consequently determine the transmittance. The transmittance peaks of incident beams with different optical angular momenta will appear at different radii of the nano grooves, which provides a potential way to modulate the transmittance by adjusting the radius of the nano groove. This study is instructive for designing the enhanced optical transmission nano device based on vortex SPP.