The phenomenon mode instability is the most limiting factor for further scaling the output power and beam quality in high power fiber lasers. Thus, it is meaningful and necessary to study the influencing factor of mode instability and finally find the approaches to mitigating its influence. Theoretical calculations reveal that the fiber V-parameter has a negative effect on fiber amplifier mode instability threshold. Nevertheless, the influence of fiber core numerical aperture (NA) on fiber oscillator mode instability threshold has rarely been investigated compared with that on the fiber amplifier. In this paper, we build a high-power all-fiber laser oscillator pumped by 976nm laser diodes and measure its laser efficiency and mode instability threshold of 20/400 step-index ytterbium doped fiber with different fiber core NA. Experimental result reveals that at the same 976 nm pump power, the fiber with relatively low core NA (~0.059) has a higher mode instability threshold power than that with relatively high core NA (~0.064), and that even a higher core NA (~0.064) fiber has a higher laser efficiency than lower core NA (~0.059) fiber. The fact shows that the fiber core NA has a significant influence on mode instability threshold, and a relatively high core NA results in a lower mode instability threshold. Also, numerical simulations explain the reason why the fiber core NA has a negative effect on mode instability threshold in fiber oscillator. First of all, the higher fiber core NA will support more propagating modes in fiber, and the lower fiber core NA will result in higher order mode (HOM) content leaking into fiber cladding and the overlap of HOM content and gain area is reduced, thus the gain of HOM is relatively reduced. Also, the bending loss of HOM is very sensitive to fiber core NA variation, and the increase of fiber core NA will reduce the bending loss of HOM dramatically. In conclusion, the fiber core NA has a significant negative effect on fiber oscillator mode instability threshold, and numerical simulationscan explain the physical origin of the negative effect of fiber core NA on laser oscillator mode instability threshold. Thus, for the mode instability mitigation in high power laser oscillator, optimizing the NA of active fiber conduces to the increase of mode instability threshold, which is helpful and necessary for further scaling the output power and beam quality.