Under the background of rural land institution reform and labor force transfer into cities, farmers are undergoing the transformation from the traditional role of agricultural production labor force to the new role of agricultural management decision makers. Given the characteristics of high cost, high labor input, and low benefit of land investment, small-scale farmers are typical risk avoiders whose land investment behavior is more sensitive to risk factors, but their cognitive mechanism and decision-making logic are not clear yet. Based on the distributed cognition theory and perceived value theory, we conducted a survey of 483 small-holder farmers in the suburbs of the Wuhan metropolitan area. Then we used the methods of structural equation model and system dynamics model to perform the static analysis and dynamic simulation of the influence of farmers’ land investment risk perception and the development trend of investment behaviors. The research shows that: (1) Farmers’ perceived risk of land investment follows the basic framework of distributed cognition theory and is affected by the three cognitive function systems of “human power” “regional power” and “cultural power”, indicating that farmers’ risk perception of land investment mainly comes from the behavior itself, which is also the result of the combined effect of individual internal factors and external environmental factors. (2) High risk and low benefits lead to weakened perceived value, which in turn lead to reduced investment willingness, then to reduced investment behavior response, and finally higher risk and lower benefits, which form a vicious circle that restrict farmers’ land investment. Under certain policy guidance and endowment constraints, farmers can maintain their land investment behavior in a short term, but the traditional inefficient land use mode will eventually be unsustainable and for small-holder farmers the decline of productive function of land resources will be irreversible, thus the goal of rural land system reform should focus on cultivating new agricultural operators and accelerating the transformation of agricultural modernization.