Fig. 1. Development of processors for matrix computation. (a) Moore’s law no longer seems applicable.¹⁶ (b) Exponential growth of energy consumption for more accurate ANN models.¹⁷ (c) Development trends of processors.¹⁸ (d) Temporal and spatial architectures for multicore parallelization.¹⁷ (e) Memristor crossbar arrays in post-Moore’s law era.¹⁹ (f) Integrated waveguide meshes for general-purpose matrix computation.²⁰

Fig. 2. Intuitive visualization showing energy-efficient models processing in the convolutional neural network. The CONV between the filters and kernel can be deployed into the MVM to improve efficiency.

Fig. 3. Integrated waveguide meshes: from QIP to MVM. (a) Bulk-optical CNOT gate in 2004.²⁹ (b) On-chip photonic CNOT gates in 2007.³⁰ (c) Programmable quantum processor in 2016.³¹ (d) Large-scale photonic processor for arbitrary two-qubit operations.³² (e) Large-scale photonic processor for multidimensional quantum entanglement.³³ (f) Schematic of optical switch topologies in the data center.³⁴ (g) Reconfigurable hexagonal mesh for programmable signal processing.³⁵ (h) Photonic “FPGA” for programmable radiofrequency signal processing.³⁶ (i) Self-configuring 4×4-port linear processor. (j) First optical computing processor for vowel recognition.²⁰ (k) Large scale 64×64 MVM processor.^37,38 (l) FFTNet architecture for better fault tolerance against imprecise components.³⁹ (m) Redundant architecture to overcome fabrication errors.⁴⁰

Fig. 4. Multiple laser source MVM. (a) MVM based on microring modulators.⁵⁴ (b) MVM based on on-chip photorefractive interaction.^55,56 (c) Photonic tensor core constituted by dot-product engines.⁵⁷ (d) Photonic crossbar arrays with phase-change material.⁵⁸

Fig. 5. FT-based CONV. (a) Flowchart of CONV using FT. (b) FT based on MMI coupler and compensating phase shifter arrays.⁵⁹ (c) FT operation with 21×21-star coupler.⁶⁰ (d) Compact surface plasmon polaritons device for FT.⁶¹ (e) CONV based on Cooley–Tukey method FT.⁶²

Fig. 6. Element-wise MAC operations. (a) Basic 3×3 CONV consisting of nine MAC operations. (b) MAC based on balanced homodyne detection.⁶³ (c) MAC based on cascaded acousto-optical modulators.⁶⁴ (d) MAC based on microring modulators.⁶⁵

Fig. 7. MAC based on dispersion. (a) The 118 GigaMAC/s matrix operation is realized by 1.1-km long linear dispersion fiber.⁶⁶ (b) 11.9 GigaFLOPs/s MAC conducted with 13-km spool of standard single-mode fiber.⁶⁷ (c) Time-stretch method for MAC operations.⁶⁸ (d) Temporal CONV (a series of MAC) by spiral waveguide with linear group dispersion.⁶⁹

Fig. 8. Interconnections in processors, memory, and peripheral hardware. (a) The memory-processor interconnections are one of the major factors influencing the overall performance and the memory wall problem that has hindered high-performance computing.⁷⁰ (b) Large-scale matrix multiplication is decomposed into small-scale matrix multiplications while processing.⁷¹ (c) On-chip optical transceivers are good alternatives for low-energy-budget interconnections and boosting the data movement among the computation hardware.⁷²

Fig. 9. Optical computing from bulk-optics to photonic-electronic integration. (a) SLM-based MVM processor released by Enlight in 2003.⁸² (b) Bulk-optical 4f-system for convolutional neural network.⁸³ (c) Diffractive deep neural network by 3D-printed multi-layer phase mask.⁸⁴ (d) 3D copackaged module for enhancing the interaction between the photonic core and electronic ASIC.³⁴

Fig. 10. Energy efficiency of silicon-based optoelectronic matrix computation processor (consider all the photonic, optoelectronic, and electronic devices and circuits). (a) The equivalent energy efficiency (energy consumption per MAC operation) linearly decreases as the side-length of the matrix increases.⁶³ (b) Expectations of future compute density and energy efficiency in silicon-based optoelectronic matrix computation (the energy efficiency depends on the matrix configuration).

Fig. 11. Photonic matrix computation can be used for solving some difficult problems and reducing their time complexity. (a) Heuristic recurrent algorithm for the annealing of Ising models. (b) Reconstruction of K-sparse signals in compressed sensing. (c) Very large-scale discrete Fourier transform.