At present, China’s food security situation is grim because cultivated land is the basis of food security, but abandoned cultivated land has become a common phenomenon of land use change. This study used the panel data of household survey in China to analyze the change of cultivated land use behavior and influencing factors of farming households that contracted cultivated land from 2010 to 2016, and examined the probability change and regional difference from the perspectives of explicit abandonment and implicit abandonment. The results show that: First, cultivated land contracted by farmers experienced both explicit and implicit abandonments. Implicit abandonment is relatively large, but the phenomenon of abandonment in the future will gradually decrease. Second, in terms of consistency, investment in seeds, pesticides, and fertilizers, and the implementation of agricultural subsidy policies have significant inhibitory effects on land abandonment, but their marginal effects will be reduced. Land expropriation will prompt farmers to choose land abandonment, which has a negative long-term effect. Third, the influencing factors of explicit and implicit abandonments are different. Explicit abandonment is significant in hilly areas. The scale of cultivated land and the strategy of returning the grain plots to forestry and grass can reduce the probability of implicit abandonment for farmers. Natural hazards will cause explicit and implicit land abandonment, the marginal effect of natural hazards on explicit abandonment will be reduced, but the marginal effect on implicit abandonment will increase. Last, in terms of spatial differences, it is still possible that the cultivated land in the economically developed areas will experience an explicit abandonment, and the main grain-producing areas may continue to experience an implicit abandonment. In the future, it is necessary to formulate and improve relevant land and agricultural policies, continue to increase agricultural investment, carry out large-scale land transfers, and so on in order to solve the problem of farmland abandonment.