Artificial intelligence (AI) is one of the most active research fields at present. The AI models, represented by artificial neural networks, are computational models that mimic the neural synaptic networks in the brain and have been widely used in areas such as computer vision, speech recognition, and automatic driving. In the last decade, the AI technologies have experienced an explosive growth and the global computational volume has increased dramatically. The urgent need to process massive data in a fast and efficient way has placed an urgent demand on the computing hardware in terms of computing capacity and energy efficiency. Restricted by the inherent limits of electronic devices and the von Neumann architecture, traditional electronic computing has encountered a bottleneck in terms of speed and energy efficiency, which is difficult to break through. Optoelectronic intelligent computing uses photons instead of electrons to perform computation, hence optoelectronic intelligent computing can significantly improve computing speed and energy efficiency by overcoming the inherent limits of electrons. Compared with electronic computing, optoelectronic intelligent computing fully combines the unique advantages of multi-dimensional multiplexing, large bandwidth, and low energy consumption of optics and the fine-grained and flexible control of electronics, which is a more practical and competitive solution for accelerating AI computing.

Optoelectronic intelligent computing is particularly suitable for implementing large-scale neural networks containing a large number of neurons and synapses. Restricted by interconnection density and Joule heat, the processing speed of current neuromorphic electronic chips is basically limited to the MHz range, and the energy consumption per multiply accumulate (MAC) operation requires several picojoules. However, the neuromorphic computing hardware built from basic photonic devices, such as Mach-Zehnder interferometer (MZI) mesh and micro-ring resonator (MRR) array, requires only tens of femtojoules per MAC operation. This metric is two orders of magnitude smaller than that of the state-of-the-art complementary metal oxide semiconductor (CMOS) computing hardware, indicating that optical neural networks are far superior to electronic neural networks in terms of energy efficiency while achieving ultra-high-speed computing. As a result, optoelectronic intelligent computing naturally has significant advantages in the application scenarios such as automatic driving and drones which require large bandwidth and high real-time performance, as well as in the highly concurrent, high-throughput, computationally intensive supercomputing platforms and data centers. In fact, the optical interconnect technique has already been widely used in data centers and significantly reduces its time and energy consumption for large-scale interconnects.

Generally speaking, AI algorithms can be divided into two parts: training and inference. Since photons are difficult to be stored and the state of photons cannot be directly obtained, most of the current optoelectronic intelligent computing systems use the method of "training in the electronic domain and inference in the optical domain" to implement the AI algorithms. In other words, the simulation model of the neural network is first trained on the electronic computer, and then the parameters of the trained model are loaded onto the photonic chip for inference. However, this offline training method obviously has the difference between the the numerical simulation model and the actual physical model, and more importantly, if the neural network implementation of the photonic chip always needs to rely on the training in the electronic domain, then the performance advantages of the photonic chip over the microelectronic chip including low latency and high energy efficiency cannot be fully exploited. Therefore, developing hardware-friendly online training algorithms for optoelectronic intelligent computing is a key challenge.

Here, a comprehensive review of the research progress and challenges in optoelectronic intelligent computing is presented. There are three mainstream types of optical matrix-vector multiplication (MVM), which are plane light conversion (PLC)-based matrix computing, MZI-based matrix computing, and wavelength division multiplexing (WDM)-based matrix computing (Fig. 1). The applications of optoelectronic intelligent computing mainly consist of optical signal processing and optical neural networks (Fig. 2). Hardware-friendly online training algorithms for optoelectronic intelligent computing mainly include online training of optical neural networks through the back propagation (BP) algorithm (Fig. 3) and online training of optoelectronic intelligent computing chips through the stochastic gradient descent (SGD) algorithm (Fig. 4). Computing capacity and energy efficiency are important metrics to evaluate the performance of optoelectronic intelligent computing. Table 1 shows the comparison of computing capacity and energy efficiency of various microelectronic chips and optoelectronic chips. As for three typical optoelectronic intelligent computing architectures (Fig. 5), their computing capacity and energy efficiency are summarized (Table 2). Finally, according to the aforementioned evaluation methods, the ways to further improve computing capacity and reduce energy consumption can be explored in terms of improving parallelism, baud rate, operation scale, and optical-electrical/electrical-optical conversion efficiency and reducing energy consumption in the optical and electrical layers (Fig. 6).

Optoelectronic intelligent computing fully combines the advantages of multi-dimensional multiplexing, large bandwidth, and low energy consumption of optics and the fine-grained and flexible control of electronics. It is a category of the dedicated computing hardware architectures for AI computing, which breaks the bottleneck of traditional von Neumann architectures. Here, the research progress of optoelectronic intelligent computing is reviewed, the challenges in online training algorithms as well as computing capacity and energy efficiency improvement of the current mainstream computing architectures for optical signal processing and optical neural networks are discussed, and the perspectives are presented. Advanced optoelectronic integration and fabrication techniques enable the production of large-scale and low-cost optoelectronic computing chips. Through optoelectronic monolithic integration, electronic and photonic devices can be integrated on the same substrate, which eliminates on-chip and off-chip optoelectronic interconnections and builds on-chip optoelectronic hybrid computing architectures. In addition, the performance of optoelectronic intelligent computing can be further improved by in-depth combination between novel materials with excellent performance and customized design with hardware-software synergy.