The rapid development of information technology has fueled an ever-increasing demand for ultrafast and ultralow-energy-consumption computing. Existing computing instruments are pre-dominantly electronic processors, which use electrons as information carriers and possess von Neumann architecture featured by physical separation of storage and processing. The scaling of computing speed is limited not only by data transfer between memory and processing units, but also by RC delay associated with integrated circuits. Moreover, excessive heating due to Ohmic losses is becoming a severe bottleneck for both speed and power consumption scaling. Using photons as information carriers is a promising alternative. Owing to the weak third-order optical nonlinearity of conventional materials, building integrated photonic computing chips under traditional von Neumann architecture has been a challenge. Here, we report a new all-optical computing framework to realize ultrafast and ultralow-energy-consumption all-optical computing based on convolutional neural networks. The device is constructed from cascaded silicon Y-shaped waveguides with side-coupled silicon waveguide segments which we termed “weight modulators” to enable complete phase and amplitude control in each waveguide branch. The generic device concept can be used for equation solving, multifunctional logic operations as well as many other mathematical operations. Multiple computing functions including transcendental equation solvers, multifarious logic gate operators, and half-adders were experimentally demonstrated to validate the all-optical computing performances. The time-of-flight of light through the network structure corresponds to an ultrafast computing time of the order of several picoseconds with an ultralow energy consumption of dozens of femtojoules per bit. Our approach can be further expanded to fulfill other complex computing tasks based on non-von Neumann architectures and thus paves a new way for on-chip all-optical computing.