• Advanced Photonics
  • Vol. 6, Issue 2, 024001 (2024)
Wenhao Tang1、†, Qing Yang1、2、3, Hang Xu1, Yiyu Guo1, Jiqiang Zhang1, Chunfang Ouyang4, Leixin Meng1、*, and Xu Liu2、3、*
Author Affiliations
  • 1Zhejiang Laboratory, Research Center for Frontier Fundamental Studies, Hangzhou, China
  • 2Zhejiang University, College of Optical Science and Engineering, State Key Laboratory of Extreme Photonics and Instrumentation, Hangzhou, China
  • 3ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, China
  • 4Shanghai Jiao Tong University, Chip Hub for Integrated Photonics Xplore (CHIPX), Wuxi, China
  • show less
    DOI: 10.1117/1.AP.6.2.024001 Cite this Article Set citation alerts
    Wenhao Tang, Qing Yang, Hang Xu, Yiyu Guo, Jiqiang Zhang, Chunfang Ouyang, Leixin Meng, Xu Liu. Review of bio-inspired image sensors for efficient machine vision[J]. Advanced Photonics, 2024, 6(2): 024001 Copy Citation Text show less
    References

    [1] Y. Chai. In-sensor computing for machine vision. Nature, 579, 32-33(2020).

    [2] S. Dhawan. A review of image compression and comparison of its algorithms. Int. J. Electron. Commun. Technol., 2, 22-26(2011).

    [3] F. Mentzer et al. High-fidelity generative image compression, 11913-11924(2020).

    [4] L. C. Ngugi et al. Recent advances in image processing techniques for automated leaf pest and disease recognition–a review. Inf. Process. Agric., 8, 27-51(2021).

    [5] M. J. Weinberger et al. The LOCO-I lossless image compression algorithm: principles and standardization into JPEG-LS. IEEE Trans. Image Process., 9, 1309-1324(2000).

    [6] X. Pitkow et al. Decorrelation and efficient coding by retinal ganglion cells. Nat. Neurosci., 15, 628-635(2012).

    [7] M. F. Bear et al. Synaptic plasticity: LTP and LTD. Curr. Opin. Neurobiol., 4, 389-399(1994).

    [8] T. Hosoya et al. Dynamic predictive coding by the retina. Nature, 436, 71-77(2005).

    [9] T. Gollisch. Throwing a glance at the neural code: rapid information transmission in the visual system. HFSP J., 3, 36-46(2009).

    [10] T. Gollisch et al. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron, 65, 150-164(2010).

    [11] B. A. Olshausen et al. Sparse coding of sensory inputs. Curr. Opin. Neurobiol., 14, 481-487(2004).

    [12] T.-H. Hsu et al. AI edge devices using computing-in-memory and processing-in-sensor: from system to device, 22.25.1-22.25.4(2019).

    [13] K. D. Choo et al. Energy-efficient motion-triggered IoT CMOS image sensor with capacitor array-assisted charge-injection SAR ADC. IEEE J. Solid-State Circuit, 54, 2921-2931(2019).

    [14] Z. Du et al. ShiDianNao: shifting vision processing closer to the sensor, 92-104(2015).

    [15] A. Jimenez-Fernandez et al. A binaural neuromorphic auditory sensor for FPGA: a spike signal processing approach. IEEE Trans. Neural Netw. Learn Syst., 28, 804-818(2017).

    [16] L. M. Chalupa, P. Sterling, J. S. Werner. How retinal circuits optimize the transfer of visual information. The Visual Neurosciences, 234-259(2004).

    [17] J. J. O’Brien et al. Photoreceptor coupling mediated by connexin36 in the primate retina. J. Neurosci., 32, 4675-4687(2012).

    [18] Z. Yu et al. Toward the next generation of retinal neuroprosthesis: visual computation with spikes. Engineering, 6, 449-461(2020).

    [19] N. Kruger et al. Deep hierarchies in the primate visual cortex: what can we learn for computer vision?. IEEE Trans. Pattern Anal. Mach. Intell., 35, 1847-1871(2012).

    [20] M. Kiselev. Rate coding vs. temporal coding:is optimum between?, 1355-1359(2016).

    [21] M. N. Shadlen et al. Noise, neural codes and cortical organization. Curr. Opin. Neurobiol., 4, 569-579(1994).

    [22] J. Benda. Neural adaptation. Curr. Opin. Neurobiol., 31, R110-R116(2021).

    [23] D. Lee et al. In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing. Nat. Commun., 13, 5223(2022).

    [24] M. Kim et al. DeepPep: deep proteome inference from peptide profiles. PLoS Comput. Biol., 13, e1005661(2017).

    [25] C. Y. Fong et al. Auditory mismatch negativity under predictive coding framework and its role in psychotic disorders. Front. Psychiatry, 11, 557932(2020).

    [26] N. Waltham. CCD and CMOS Sensors(2013).

    [27] A. Rodríguez-Vázquez et al. CMOS vision sensors: embedding computer vision at imaging front-ends. IEEE Circuits Syst. Mag., 18, 90-107(2018).

    [28] V. Shirmohammadli et al. A neuromorphic electrothermal processor for near‐sensor computing. Adv. Mater. Technol., 7, 2200361(2022).

    [29] M. Nazhamaiti et al. NS-MD: near-sensor motion detection with energy harvesting image sensor for always-on visual perception. IEEE Trans. Circuits Syst. II: Express Briefs, 68, 3078-3082(2021).

    [30] R. Forchheimer et al. Near-sensor image processing: a new paradigm. IEEE Trans. Image Process., 3, 736-746(1994).

    [31] Z. Chen et al. Processing near sensor architecture in mixed-signal domain with CMOS image sensor of convolutional-kernel-readout method. IEEE Trans. Circuits Syst. I: Regular Pap., 67, 389-400(2020).

    [32] Z. Liu et al. NS-CIM: a current-mode computation-in-memory architecture enabling near-sensor processing for intelligent IoT vision nodes. IEEE Trans. Circuits Syst. I: Regular Pap., 67, 2909-2922(2020).

    [33] R. Wang et al. Bio-inspired in-sensor compression and computing based on phototransistors. Small, 18, e2201111(2022).

    [34] Z. Zhang et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol., 17, 27-32(2022).

    [35] W. Pan et al. A future perspective on in-sensor computing. Engineering, 14, 19-21(2022).

    [36] H. Xu et al. Senputing: an ultra-low-power always-on vision perception chip featuring the deep fusion of sensing and computing. IEEE Trans. Circuits Syst. I: Regular Pap., 69, 232-243(2022).

    [37] D. Ielmini et al. In-memory computing with resistive switching devices. Nat. Electron., 1, 333-343(2018).

    [38] L. Tong et al. 2D materials-based homogeneous transistor-memory architecture for neuromorphic hardware. Science, 373, 1353-1358(2021).

    [39] L. Tong et al. Stable mid-infrared polarization imaging based on quasi-2D tellurium at room temperature. Nat. Commun., 11, 2308(2020).

    [40] J. Zha et al. Infrared photodetectors based on 2D materials and nanophotonics. Adv. Funct. Mater., 32, 2111970(2022).

    [41] J. K. Han et al. A review of artificial spiking neuron devices for neural processing and sensing. Adv. Funct. Mater., 32, 2204102(2022).

    [42] J. W. Han et al. Leaky integrate-and-fire biristor neuron. IEEE Electron Device Lett., 39, 1457-1460(2018).

    [43] J. K. Han et al. Mimicry of excitatory and inhibitory artificial neuron with leaky integrate-and-fire function by a single MOSFET. IEEE Electron Device Lett., 41, 208-211(2020).

    [44] L. Gao et al. NbOx based oscillation neuron for neuromorphic computing. Appl. Phys. Lett., 111, 103503(2017). https://doi.org/10.1063/1.4991917

    [45] D. Lee et al. Various threshold switching devices for integrate and fire neuron applications. Adv. Electron. Mater., 5, 1800866(2019).

    [46] X. Zhang et al. An artificial neuron based on a threshold switching memristor. IEEE Electron Device Lett., 39, 308-311(2018).

    [47] T. Tuma et al. Stochastic phase-change neurons. Nat. Nanotechnol., 11, 693-699(2016).

    [48] A. Sengupta et al. Magnetic tunnel junction mimics stochastic cortical spiking neurons. Sci. Rep., 6, 30039(2016).

    [49] L. Sun et al. Bio-inspired vision and neuromorphic image processing using printable metal oxide photonic synapses. ACS Photonics, 10, 242-252(2022).

    [50] L. Sun et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv., 7, eabg1455(2021).

    [51] D. Gehrig et al. Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. IEEE Rob. Autom. Lett., 6, 2822-2829(2021).

    [52] T. Delbruck et al. Utility and feasibility of a center surround event camera, 381-385(2022).

    [53] F. Liao et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat. Electron., 5, 84-91(2022).

    [54] C. Jin et al. Artificial vision adaption mimicked by an optoelectrical In2O3 transistor array. Nano Lett., 22, 3372-3379(2022). https://doi.org/10.1021/acs.nanolett.2c00599

    [55] S.-C. Liu et al. Event-driven sensing for efficient perception: vision and audition algorithms. IEEE Signal Process. Mag., 36, 29-37(2019).

    [56] A. Vitale et al. Event-driven vision and control for UAVs on a neuromorphic chip, 103-109(2021).

    [57] H. Rebecq et al. EMVS: event-based multi-view stereo—3D reconstruction with an event camera in real-time. Int. J. Comput. Vis., 126, 1394-1414(2018).

    [58] L. Zhu et al. Hybrid coding of spatiotemporal spike data for a bio-inspired camera. IEEE Trans. Circuits Syst. Video Technol., 31, 2837-2851(2020).

    [59] S. Dong et al. Spike coding for dynamic vision sensor in intelligent driving. IEEE Internet Things J., 6, 60-71(2019).

    [60] K. A. Zaghloul et al. A silicon retina that reproduces signals in the optic nerve. J. Neural Eng., 3, 257(2006).

    [61] J. Costas-Santos et al. A spatial contrast retina with on-chip calibration for neuromorphic spike-based AER vision systems. IEEE Trans. Circuits Syst. I: Regular Pap., 54, 1444-1458(2007).

    [62] J. A. Lenero-Bardallo et al. A signed spatial contrast event spike retina chip, 2438-2441(2010).

    [63] F. Zhou et al. Near-sensor and in-sensor computing. Nat. Electron., 3, 664-671(2020).

    [64] P. M. Sheridan et al. Sparse coding with memristor networks. Nat. Nanotechnol., 12, 784-789(2017).

    [65] V. K. Sangwan et al. Multi-terminal memtransistors from polycrystalline monolayer molybdenum disulfide. Nature, 554, 500-504(2018).

    [66] V. K. Sangwan et al. Neuromorphic nanoelectronic materials. Nat. Nanotechnol., 15, 517-528(2020).

    [67] Y. Yuan et al. Ion migration in organometal trihalide perovskite and its impact on photovoltaic efficiency and stability. Accounts Chem. Res., 49, 286-293(2016).

    [68] A. M. Leguy et al. The dynamics of methylammonium ions in hybrid organic–inorganic perovskite solar cells. Nat. Commun., 6, 7124(2015).

    [69] J. You et al. Low-temperature solution-processed perovskite solar cells with high efficiency and flexibility. ACS Nano, 8, 1674-1680(2014).

    [70] B. J. Kim et al. Highly efficient and bending durable perovskite solar cells: toward a wearable power source. Energy Environ. Sci., 8, 916-921(2015).

    [71] S. F. Leung et al. A self‐powered and flexible organometallic halide perovskite photodetector with very high detectivity. Adv. Mater., 30, 1704611(2018).

    [72] V. K. Hsiao et al. Photo-carrier extraction by triboelectricity for carrier transport layer-free photodetectors. Nano Energy, 65, 103958(2019).

    Wenhao Tang, Qing Yang, Hang Xu, Yiyu Guo, Jiqiang Zhang, Chunfang Ouyang, Leixin Meng, Xu Liu. Review of bio-inspired image sensors for efficient machine vision[J]. Advanced Photonics, 2024, 6(2): 024001
    Download Citation