Convolutional neural networks (CNNs) have seen tremendous success in many applications, such as speech and image recognition [1,2], computer vision [3], and document analysis [4]. However, CNN-based systems are computationally expensive due to their complicated architectures and the large number of parameters they rely on. CNNs therefore typically require the implementation of multicore central processing units and graphics processing units to compensate for the rather high computational expense [5,6]. This makes CNN architectures often unsuitable for smaller devices like phones and smart cameras, where power and speed have strict limitations. To address these drawbacks, the optimization and discovery of new high-speed and low power consumption platforms for CNNs are urgently required. For the optimization of CNNs, binary CNNs, which are simple, efficient, and accurate approximations of complete CNNs, can be introduced [7–9]. In binary CNNs, the weights given to the inputs of each convolutional layer are approximated with binary values [7]. Therefore, binary CNNs boast $58\times$ faster convolutional operations and $32\times$ less memory requirements than traditional CNNs [7]. Several optimized binary versions of CNNs have been proposed for training processes and image classification tasks [7,10,11]. However, beyond the optimization of CNNs, a new platform offering high speed and low power consumption remains highly desirable.

Photonics is considered a highly promising candidate for future neural network implementations given the unique advantages it provides such as high speed, wide bandwidth, and low power consumption [12–21]. Photonics-based CNNs have therefore been proposed in order to increase the speed of convolutional operations [18–21]. A photonic CNN accelerator was proposed based on silicon photonic micro-ring weighting banks [18]. The full system design offers more than three orders of magnitude improvement in execution time, and its optical core potentially offers more than five orders of magnitude improvement compared to state-of-the-art electronic counterparts [18]. Xu et al. also proposed high-accuracy optical convolution unit architecture based on acousto-optical modulator arrays, where the optical convolution unit was shown to perform well on inferences of typical CNN tasks [20]. However, the size of the system is based on the size of the kernel utilized in these emerging works on photonic CNNs.

In this work, we propose an all-optical binary convolution system using a single vertical-cavity surface-emitting laser (VCSEL) operating as a spiking optical neuron, hence, dramatically reducing hardware requirements. In our approach, temporal encoding is used instead of spatial encoding, thus crucially helping to reduce (optical) hardware complexity. In our all-optical binary convolution technique, results are represented by the number of fast ($<100\mathrm{ps}$ long) spiking responses delivered by the optical spiking VCSEL neuron. This has unique advantages in terms of robustness to noise and high precision. Additionally, VCSELs have unique inherent advantages, such as high-energy efficiency, high-speed modulation capability, low bias currents, easy packaging, and highly integrable structures [22,23]. In particular, VCSELs have demonstrated the ability to generate fast spiking dynamics analogous to those of biological neurons known for their robustness to input noise [24–29]. The controlled activation, inhibition, and communication of these neuronal dynamics has been demonstrated, and recently a single VCSEL device was used to perform spiking pattern recognition and rate coding [24–31]. Thus, photonic spiking VCSELs make suitable candidates for a new future photonic platform for ultrafast energy efficient spiking CNNs.

In this work, we use a VCSEL-based photonic approach for binary convolution to demonstrate image gradient magnitude calculation. This delivers an essential portion of the image edge detection functionality used by computer vision and image recognition systems. Here, a single VCSEL system is developed to solely perform a convolution operation; hence, no VCSEL-based CNN architecture, capable of providing learning and classification capabilities, is discussed in this work. The rest of the paper is organized as follows. Section 2 is devoted to the experimental setup of this work for the demonstration of all-optical binary convolution with a spiking VCSEL neuron and the theoretical model used to predict the response of the system. In Section 3, convolutional results are analyzed before the full calculation of image gradient magnitudes is performed both experimentally and theoretically. Finally, Section 4 summarizes the conclusions of this work.

We present here the experimental arrangement and theoretical model of the all-optical binary convolution system based on a photonic spiking VCSEL neuron. In this work, we set a source digital image and a kernel as the two inputs of the convolution system. The value of any one pixel in the source image or kernel is limited to 0 or 1.

We use an extension of the well-known spin-flip model (SFM) to model the operation of the VCSEL acting as a spiking optical neuron. In our formulation, we add extra terms to the model’s equations to account for the source image and kernel inputs. The rate equations can be described as follows [26,27]: $$\begin{gathered} \frac{\mathrm{d}{E}_{x,y}}{\mathrm{d}t}=-(k\pm {\gamma }_a)E_{x,y}-i(k\alpha \pm {\gamma }_p)E_{x,y}+k(1+i\alpha )(N{E}_{x,y} \\ \pm in{E}_{y,x})+k_{\mathrm{inj}}[E_{\text{inj}x1}(t)+E_{\text{inj}x2}(t)]e^{i\mathrm{\Delta }{\omega }_{x}t}+F_{x,y}, \end{gathered}$$(1)$$\frac{\mathrm{d}N}{\mathrm{d}t}=-{\gamma }_N[N(1+{|E_x|}^2+{|E_y|}^2)-\mu +in(E_y{E}_x^{*}-E_x{E}_y^{*})],$$(2)$$\frac{\mathrm{d}n}{\mathrm{d}t}=-{\gamma }_{s}n-{\gamma }_N[n({|E_x|}^2+{|E_y|}^2)+iN(E_y{E}_x^{*}-E_x{E}_y^{*})],$$(3)where the subscripts $x$, $y$ represent the XP and YP modes of the VCSEL, respectively. $E_{x,y}$ is the slowly varying complex amplitude of the field in the XP and YP modes. $N$ is the total carrier inversion between conduction and valence bands. $n$ is the difference between carrier inversions with opposite spins. $k$ denotes the field decay rate. ${\gamma }_a$ and ${\gamma }_p$ are the linear dichroism and the birefringence rate, respectively. $\alpha$ is the linewidth enhancement factor. ${\gamma }_N$ is the decay rate of $N$. ${\gamma }_s$ is the spin-flip rate. $\mu$ represents the normalized pump current. $k_{\mathrm{inj}}$ is the injected strength, and $E_{\text{inj}x1}$ and $E_{\text{inj}x2}$ indicate, respectively, the source image and kernel inputs. $\mathrm{\Delta }{\omega }_x$ is defined as $\mathrm{\Delta }{\omega }_x={\omega }_{\text{inj}x}-{\omega }_0$, where ${\omega }_{\text{inj}x}$ is the angular frequency of the externally injected light in the XP mode, and ${\omega }_0=({\omega }_x+{\omega }_y)/2$ is the center frequency between the XP and YP modes with ${\omega }_x={\omega }_0+\alpha {\gamma }_a-{\gamma }_p$ and ${\omega }_y={\omega }_0-\alpha {\gamma }_a+{\gamma }_p$. The frequency detuning between the externally injected signal and the XP mode is set as $\mathrm{\Delta }{f}_x=f_{\text{inj}x}-f_x$. Hence, in Eq. (1), $\mathrm{\Delta }{\omega }_x=2\pi \mathrm{\Delta }{f}_x+\alpha {\gamma }_a-{\gamma }_p$. $F_{x,y}$ are the spontaneous emission noise terms, which can be written as $$F_x=\sqrt{\frac{{\beta }_{\mathrm{sp}}{\gamma }_N}{2}}(\sqrt{N+n}{\xi }_1+\sqrt{N-n}{\xi }_2),$$(4)$$F_y=-i\sqrt{\frac{{\beta }_{\mathrm{sp}}{\gamma }_N}{2}}(\sqrt{N+n}{\xi }_1-\sqrt{N-n}{\xi }_2),$$(5)where ${\beta }_{\mathrm{sp}}$ is the strength of the spontaneous emission, and ${\xi }_1$ and ${\xi }_2$ are independent complex Gaussian white noise terms of zero mean and a unit variance. We numerically solve Eqs. (1)–(4) using the fourth-order Runge–Kutta method. The parameter values configured for the 1300 nm VCSEL are as follows [27]: $k=185{\mathrm{ns}}^{-1}$, ${\gamma }_a=2{\mathrm{ns}}^{-1}$, ${\gamma }_p=128{\mathrm{ns}}^{-1}$, $\alpha =2$, ${\gamma }_N=0.5{\mathrm{ns}}^{-1}$, ${\gamma }_s=110{\mathrm{ns}}^{-1}$, ${\beta }_{\mathrm{sp}}={10}^{-6}$, and $k_{\text{inj}}=125{\mathrm{ns}}^{-1}$. With these parameters, the YP mode is the main lasing mode, and the XP mode is the subsidiary mode, as in Fig. 2(a).

In this section, we firstly provide an experimental proof-of-concept demonstration of all-optical binary convolution with a spiking VCSEL neuron. We then calculate the image gradient magnitudes from a basic “Square” source image and a complex “Horse head” source image by means of all-optical binary convolution. Simulation results on the binary convolution and the calculation of image gradient magnitudes are also presented using a “Horse” source image from the latest version of the Berkeley Segmentation Data Set [32]. Finally, the robustness of our binary convolution system is also tested numerically by adding noise to the source image and kernel inputs.

In this section, the image gradient magnitude, critical to image edge detection, is calculated using our approach based on a single spiking VCSEL neuron and optical binary convolution. The image gradient magnitude $G(x)$ of a given pixel $x$ is calculated using the following equations [33]: $$G(x)=\sqrt{G_X{(x)}^2+G_Y{(x)}^2},$$(6)$$G_X(x)=[B(x)\otimes B_X^{+}]-[B(x)\otimes B_X^{-}],$$(7)$$G_Y(x)=[B(x)\otimes B_Y^{+}]-[B(x)\otimes B_Y^{-}].$$(8)

Four binary convolutions, i.e., $B(x)\otimes B_{X,Y}^{\pm }$, are used in $G_X(x)$ and $G_Y(x)$. $B(x)=\sum _{p=0}^{N-1}s(i_p,i_x)·2^p$ is the N bit local binary pattern descriptor of pixel $x$. $i_x$ is the central pixel intensity, and $i_p$ is the intensity of the $p\mathrm{t}\mathrm{h}$ neighbor of $x$ in the source pattern. The comparison operator is defined as $$\mathit{s}(i_p,i_x)=\{\begin{array}{cc}1& \text{if}|i_p-i_x|>T_x\\ 0& \text{otherwise}\end{array},$$(9)where $T_x=\frac{1}{4}{i}_x+20$ and $N=5\times 5-1$.

For the red-highlighted pixel $x$ in Fig. 6(b), “1” in $B(x)$ corresponds to a white pixel, and “0” corresponds to a black pixel in the source image.

In Eqs. (7) and (8), $B_X^{+}$, $B_X^{-}$, $B_Y^{+}$, and $B_Y^{-}$ are the four kernels that are adopted as in Ref. [33]. Figure 6(c) shows the areas of the four different kernels. Pixels that fall outside of the highlighted areas in Fig. 6(c) for a given string are set to zero. For example, $$B_X^{+}=\left[\begin{array}{ccccc}1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 1& 1& 0& 0& 0\\ 0& 1& 0& 0& 0\\ 1& 0& 0& 0& 0\end{array}\right].$$(11)

In this section, binary convolution based on a single VCSEL neuron is performed numerically. The robustness of the system to perform all-optical binary convolution under noisy inputs and for larger kernels is investigated. Finally, the calculation of image gradient magnitudes with our photonic approach using a single VCSEL neuron is presented numerically using the “Horse” image from the latest version of the Berkeley Segmentation Data Set [32].

Optical binary convolution can be used in systems where simplified convolutional operations, with binary inputs (still able to provide high-performance accuracy), provide other key advantages in terms of increased operation speed, lowered energy consumption, and reduced hardware requirements. This is the case of the system reported in this work, using an extremely hardware friendly implementation of a single VCSEL to perform high-speed and low energy ($<\mathrm{pJ}/\text{spike}$) image edge-feature detection. Besides, in our approach, the results of the binary convolution are output in an optical spiking representation, providing unique advantages in terms of robustness to noise and high precision of convolutional results. This spiking representation therefore enables our platform to successfully perform with noisy optical and electronic signals. Whilst other recent works have recently reported complex systems for optical convolution operation using temporally modulated inputs and weights for image processing tasks showing excellent performance [36], our technique benefits from an extremely simple architecture using just one off-the-shelf, inexpensive 1300 nm VCSEL to perform the binary convolution operation for image edge-feature detection. Our approach combining a VCSEL-based spiking photonic neuron with time multiplexing is able to deliver the operation of a full neuronal layer, where each 1.5-ns-long time slot operates in fact as a virtual neuron (or node) processing specific image pixel information. This offers great promise for future implementation of interconnected VCSEL-based neuronal network architectures for image processing tasks of increased complexity (e.g.,  image classification) and using neuron-like spiking signals to operate.

The utilization of binary convolution in the calculation of image gradient maps has been reported to outperform the alternative Canny implementation [35] of image gradient maps convolution, in the Intel i7 mobile processor [33]. In that report, the frequency of the binary gradient-based edge detector was 4.7 Hz, while the Canny convolution approach was found to operate at 0.5 Hz. This indicates that binary convolution can be performed at speeds faster than alternative convolution approaches. Additionally, mobile processors operate with powers of several watts (for example, the Intel i7 has a power of 15 W) [37], whilst VCSELs, such as the one used in this work, provide low power performance typically at milliwatt (mW) and sub-mW power levels. Hence, the energy consumption for the calculation of image gradient maps obtained with our VCSEL-based optical binary convolution system can be significantly more energy efficient, as well as yield faster operation speeds, than the performance achieved with traditional or binary convolution methods in digital processors.

In this work, we proposed and investigated experimentally and numerically an all-optical binary convolution system using a VCSEL operating as a photonic spiking neuron. The inputs (image and kernel) are encoded temporally using fast rectangular pulses (1.5-ns-long) and optically injected into the VCSEL neuron. The latter’s optical output directly provides the results of the convolution in the number of (sub-ns long) spikes fired. In addition to performing all-optical binary convolution, we demonstrated experimentally and numerically the ability of the proposed system to calculate the image gradient magnitudes from digital source images. This feature was successfully used to identify key edge features from a source image as well as its separate horizontal and vertical components. Furthermore, we investigated numerically the robustness of the proposed VCSEL-based convolutional system to input noise. This simple system, using a single commercially available VCSEL operating at the key telecom wavelength of 1300 nm, offers a novel photonic solution to binary convolution with the advantage of being highly energy efficient and hardware friendly. This opens exciting prospects for a new photonic spiking platform for future optical binary spiking CNNs. Furthermore, the high-speed, low cost, and neuronal functionalities of these photonic spiking systems hold promise for numerous processing tasks expanding into fields such as computer vision and artificial intelligence.

Acknowledgment. We thank Prof. T. Ackemann and Prof. A. Kemp (University of Strathclyde) for lending some of the equipment used in this work. All data underpinning this publication are openly available from the University of Strathclyde KnowledgeBase at https://doi.org/10.15129/51af27bc-8cc2-46a7-9088-1aad87e4340c.