Systematically revealing the impact of cultivated land fragmentation (CLF) on the geographical agglomeration pattern of agricultural specialization (AS) has positive significance for national agricultural production management. Based on the data of the second national land survey and agricultural production, this study has explored the impact of CLF on spatial heterogeneity of agricultural agglomeration in China by comprehensively using the Theil index, ordinary least square model and geographically weighted regression. Results showed that: (1) the regional differentiation of the CLF in China is obvious, and the cultivated land fragmentation index is generally characterized by increasing pattern from northwest to southeast. (2) Spatially, the development level of AS in China has formed three high-value clusters in the Northeast China Plain, the Qinghai-Tibet Plateau, and the middle of the Middle-lower Yangtze Plain; and the low-value contiguous areas centered on the Yunnan-Guizhou Plateau and the Sichuan Basin and surrounding regions, with significant spatial differences. The contribution of grain crops, economic crops, and vegetables and melon to the level of AS was 74.63%, 9.09%, and 16.28%, respectively, and the pattern of agricultural geographical aggregation dominated by grain crops has primarily taken in shape. (3) CLF is significantly negatively correlated with AS, and every 1% increase in the degree of CLF will result in a decrease of about 0.2% in AS. However, the impact of CLF on the geographic agglomeration of different crop categories or groups varies significantly. Among them, CLF has a prominent impact on the specialization level of grain crops and vegetables and melon. Each 1% increase in the CLF will reduce the specialization level of grain crops by 0.38%, and increase the level of vegetables and melon by about 0.22%. (4) According to the landscape characteristics of cultivated land, the degree of spatial division and agglomeration of cultivated land patches have a significant impact on the formation of geographical agglomeration pattern of AS, and the intensity and direction of influence show significant regional differentiation, while the patch size has no significant impact.