All-optical devices, which are utilized to process optical signals without electro-optical conversion, play an essential role in the next generation ultrafast, ultralow power-consumption optical information processing systems. To satisfy the performance requirement, nonlinear optical materials that are associated with fast response, high nonlinearity, broad wavelength operation, low optical loss, low fabrication cost, and integration compatibility with optical components are required. Graphene is a promising candidate, particularly considering its electrically or optically tunable optical properties, ultrafast large nonlinearity, and high integration compatibility with various nanostructures. Thus far, three alloptical modulation systems utilize graphene, namely freespace modulators, fiber-based modulators, and on-chip modulators. This paper aims to provide a broad view of state-of-the-art researches on the graphene-based alloptical modulation systems. The performances of different devices are reviewed and compared to present a comprehensive analysis and perspective of graphene-based alloptical modulation devices.