In situ optical backpropagation training of diffractive optical neural networks

Tiankuang Zhou; Lu Fang; Tao Yan; Jiamin Wu; Yipeng Li; Jingtao Fan; Huaqiang Wu; Xing Lin; Qionghai Dai

doi:10.1364/PRJ.389553

[1] Y. LeCun, Y. Bengio, G. Hinton. Deep learning. Nature, 521, 436-444(2015).

[2] A. Krizhevsky, I. Sutskever, G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 1097-1105(2012).

[3] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process. Mag., 29, 82-97(2012).

[4] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot. Mastering the game of go with deep neural networks and tree search. Nature, 529, 484-489(2016).

[5] A. Esteva, A. Robicquet, B. Ramsundar, V. Kuleshov, M. DePristo, K. Chou, C. Cui, G. Corrado, S. Thrun, J. Dean. A guide to deep learning in healthcare. Nat. Med., 25, 24-29(2019).

[6] G. Barbastathis, A. Ozcan, G. Situ. On the use of deep learning for computational imaging. Optica, 6, 921-943(2019).

[7] A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, M. Bethge. DeepLabCut: Markerless Pose Estimation of User-Defined Body Parts with Deep Learning(2018).

[8] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, C. J. Pal. Deep complex networks(2017).

[9] K. He, X. Zhang, S. Ren, J. Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770-778(2016).

[10] P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada, F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345, 668-673(2014).

[11] J. Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu, W. He. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 572, 106-111(2019).

[12] B. Marr, B. Degnan, P. Hasler, D. Anderson. Scaling energy per operation via an asynchronous pipeline. IEEE Trans. Very Large Scale Integr. Syst., 21, 147-151(2012).

[13] J. M. Shainline, S. M. Buckley, R. P. Mirin, S. W. Nam. Superconducting optoelectronic circuits for neuromorphic computing. Phys. Rev. Appl., 7, 034013(2017).

[14] P. R. Prucnal, B. J. Shastri. Neuromorphic Photonics(2017).

[15] D. Woods, T. J. Naughton. Optical computing: photonic neural networks. Nat. Phys., 8, 257-259(2012).

[16] D. R. Solli, B. Jalali. Analog optical computing. Nat. Photonics, 9, 704-706(2015).

[17] Q. Zhang, H. Yu, M. Barbiero, B. Wang, M. Gu. Artificial neural networks enabled by nanophotonics. Light Sci. Appl., 8, 1(2019).

[18] X. Luo. Engineering optics 2.0: a revolution in optical materials, devices, and systems. ACS Photon., 5, 4724-4738(2018).

[19] P. Minzioni, C. Lacava, T. Tanabe, J. Dong, X. Hu, G. Csaba, W. Porod, G. Singh, A. E. Willner, A. Almaiman. Roadmap on all-optical processing. J. Opt., 21, 063001(2019).

[20] Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund. Deep learning with coherent nanophotonic circuits. Nat. Photonics, 11, 441-446(2017).

[21] T. W. Hughes, R. J. England, S. Fan. Reconfigurable photonic circuit for controlled power delivery to laser-driven accelerators on a chip. Phys. Rev. Appl., 11, 064014(2019).

[22] X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, A. Ozcan. All-optical machine learning using diffractive deep neural networks. Science, 361, 1004-1008(2018).

[23] T. Yan, J. Wu, T. Zhou, H. Xie, F. Xu, J. Fan, L. Fang, X. Lin, Q. Dai. Fourier-space diffractive deep neural network. Phys. Rev. Lett., 123, 023901(2019).

[24] G. Van der Sande, D. Brunner, M. C. Soriano. Advances in photonic reservoir computing. Nanophotonics, 6, 561-576(2017).

[25] L. Larger, A. Baylón-Fuentes, R. Martinenghi, V. S. Udaltsov, Y. K. Chembo, M. Jacquot. High-speed photonic reservoir computing using a time-delay-based architecture: million words per second classification. Phys. Rev. X, 7, 011015(2017).

[26] J. Feldmann, N. Youngblood, C. Wright, H. Bhaskaran, W. Pernice. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature, 569, 208-214(2019).

[27] R. Hamerly, L. Bernstein, A. Sludds, M. Soljačić, D. Englund. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X, 9, 021032(2019).

[28] I. Chakraborty, G. Saha, K. Roy. Photonic in-memory computing primitive for spiking neural networks using phase-change materials. Phys. Rev. Appl., 11, 014063(2019).

[29] T. Deng, J. Robertson, Z.-M. Wu, G.-Q. Xia, X.-D. Lin, X. Tang, Z.-J. Wang, A. Huartado. Stable propagation of inhibited spiking dynamics in vertical-cavity surface-emitting lasers for neuromorphic photonic networks. IEEE Access, 6, 67951-67958(2018).

[30] J. Robertson, T. Deng, J. Javaloyes, A. Hurtado. Controlled inhibition of spiking dynamics in VCSELs for neuromorphic photonics: theory and experiments. Opt. Lett., 42, 1560-1563(2017).

[31] T. W. Hughes, I. A. Williamson, M. Minkov, S. Fan. Wave physics as an analog recurrent neural network(2019).

[32] J. Bueno, S. Maktoobi, L. Froehly, I. Fischer, M. Jacquot, L. Larger, D. Brunner. Reinforcement learning in a large-scale photonic recurrent neural network. Optica, 5, 756-760(2018).

[33] E. Khoram, A. Chen, D. Liu, L. Ying, Q. Wang, M. Yuan, Z. Yu. Nanophotonic media for artificial neural inference. Photon. Res., 7, 823-827(2019).

[34] A. S. Backer. Computational inverse design for cascaded systems of metasurface optics(2019).

[35] S. Maktoobi, L. Froehly, L. Andreoli, X. Porte, M. Jacquot, L. Larger, D. Brunner. Diffractive coupling for photonic networks: how big can we go?. IEEE J. Sel. Top. Quantum Electron., 26, 7600108(2019).

[36] Y. Zuo, B. Li, Y. Zhao, Y. Jiang, Y.-C. Chen, P. Chen, G.-B. Jo, J. Liu, S. Du. All optical neural network with nonlinear activation functions. Optica, 6, 1132-1137(2019).

[37] J. Chang, V. Sitzmann, X. Dun, W. Heidrich, G. Wetzstein. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep., 8, 12324(2018).

[38] H. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, A. Molnar. ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels. IEEE Conference on Computer Vision and Pattern Recognition, 903-912(2016).

[39] Y. Luo, D. Mengu, N. T. Yardimci, Y. Rivenson, M. Veli, M. Jarrahi, A. Ozcan. Design of task-specific optical systems using broadband diffractive neural networks(2019).

[40] J. Chang, G. Wetzstein. Deep optics for monocular depth estimation and 3D object detection(2019).

[41] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86, 2278-2324(1998).

[42] T. W. Hughes, M. Minkov, Y. Shi, S. Fan. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica, 5, 864-871(2018).

[43] M. Hermans, J. Dambre, P. Bienstman. Optoelectronic systems trained with backpropagation through time. IEEE Trans. Neural Netw. Learn. Syst., 26, 1545-1550(2014).

[44] M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, P. Bienstman. Trainable hardware for dynamical computing using error backpropagation through physical media. Nat. Commun., 6, 6729(2015).

[45] K. Wagner, P. Demetri. Multilayer optical learning networks. Appl. Opt., 26, 5061-5076(1987).

[46] D. Psaltis, D. Brady, K. Wagner. Adaptive optical networks using photorefractive crystals. Appl. Opt., 27, 1752-1759(1988).

[47] I. Yamaguchi, T. Zhang. Phase-shifting digital holography. Opt. Lett., 22, 1268-1270(1997).

[48] O. Mendoza-Yero, G. Mínguez-Vega, J. Lancis. Encoding complex fields by using a phase-only optical element. Opt. Lett., 39, 1740-1743(2014).

[49] M. W. Matthès, P. del Hougne, J. de Rosny, G. Lerosey, S. M. Popoff. Optical complex media as universal reconfigurable linear operators. Optica, 6, 465-472(2019).

[50] A. P. Mosk, A. Lagendijk, G. Lerosey, M. Fink. Controlling waves in space and time for imaging and focusing in complex media. Nat. Photonics, 6, 283-292(2012).

[51] Y. Li, Y. Xue, L. Tian. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica, 5, 1181-1190(2018).

[52] N. Antipa, G. Kuo, R. Heckel, B. Mildenhall, E. Bostan, R. Ng, L. Waller. DiffuserCam: lensless single-exposure 3D imaging. Optica, 5, 1-9(2018).

[53] G. K. Shirmanesh, R. Sokhoyan, P. C. Wu, H. A. Atwater. Electro-optically tunable universal metasurfaces(2019).

[54] Z. Wang, C. Li, P. Lin, M. Rao, Y. Nie, W. Song, Q. Qiu, Y. Li, P. Yan, J. P. Strachan. In situ training of feed-forward and recurrent convolutional memristor networks. Nat. Mach. Intell., 1, 434-442(2019).

CLP Journals