Deep coded exposure: end-to-end co-optimization of flutter shutter and deblurring processing for general motion blur removal

Zhihong Zhang; Kaiming Dong; Jinli Suo; Qionghai Dai

doi:10.1364/PRJ.489989

[1] Y. Pei, Y. Huang, Q. Zou, X. Zhang, S. Wang. Effects of image degradation and degradation removal to CNN-based image classification. IEEE Trans. Pattern Anal. Mach. Intell., 43, 1239-1253(2021).

[2] S. Zheng, Y. Wu, S. Jiang, C. Lu, G. Gupta. Deblur-YOLO: real-time object detection with efficient blind motion deblurring. International Joint Conference on Neural Networks (IJCNN), 1-8(2021).

[3] Q. Guo, W. Feng, R. Gao, Y. Liu, S. Wang. Exploring the effects of blur and deblurring to visual object tracking. IEEE Trans. Image Process., 30, 1812-1824(2021).

[4] R. Wang, D. Tao. Recent progress in image deblurring. arXiv(2014).

[5] J. Koh, J. Lee, S. Yoon. Single-image deblurring with neural networks: a comparative survey. Comput. Vision Image Understanding, 203, 103134(2021).

[6] K. Zhang, W. Ren, W. Luo, W.-S. Lai, B. Stenger, M.-H. Yang, H. Li. Deep image deblurring: a survey. Int. J. Comput. Vis., 130, 2103-2130(2022).

[7] S. K. Nayar, M. Ben-Ezra. Motion-based motion deblurring. IEEE Trans. Pattern Anal. Mach. Intell., 26, 689-698(2004).

[8] A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion-invariant photography. ACM Trans. Graph., 27, 1-9(2008).

[9] S. McCloskey. Temporally coded flash illumination for motion deblurring. International Conference on Computer Vision (ICCV), 683-690(2011).

[10] C. Ma, Z. Liu, L. Tian, Q. Dai, L. Waller. Motion deblurring with temporally coded illumination in an LED array microscope. Opt. Lett., 40, 2281-2284(2015).

[11] S. Elmalem, R. Giryes, E. Marom. Motion deblurring using spatiotemporal phase aperture coding. Optica, 7, 1332-1340(2020).

[12] J. Lee, B. Jeon. Multi-channel image deblurring using coded flashes. Proc. SPIE, 11766, 117660C(2021).

[13] C. M. Nguyen, J. N. P. Martel, G. Wetzstein. Learning spatially varying pixel exposures for motion deblurring. IEEE International Conference on Computational Photography (ICCP), 1-11(2022).

[14] R. Raskar, A. Agrawal, J. Tumblin. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph., 25, 795-804(2006).

[15] A. Agrawal, Yi Xu. Coded exposure deblurring: optimized codes for PSF estimation and invertibility. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2066-2073(2009).

[16] A. Agrawal, R. Raskar. Optimal single image capture for motion deblurring. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2560-2567(2009).

[17] S. McCloskey. Velocity-dependent shutter sequences for motion deblurring. Computer Vision–ECCV, 309-322(2010).

[18] S. Harshavardhan, S. Gupta, K. S. Venkatesh. Flutter shutter based motion deblurring in complex scenes. Annual IEEE India Conference (INDICON), 1-6(2013).

[19] H.-G. Jeon, J.-Y. Lee, Y. Han, S. J. Kim, I. S. Kweon. Complementary sets of shutter sequences for motion deblurring. IEEE International Conference on Computer Vision (ICCV), 3541-3549(2015).

[20] G. Cui, X. Ye, J. Zhao, L. Zhu, Y. Chen. Multi-frame motion deblurring using coded exposure imaging with complementary fluttering sequences. Opt. Laser Technol., 126, 106119(2020).

[21] R. Gonzalez, R. Woods. Digital Image Processing(2017).

[22] S. McCloskey, Y. Ding, J. Yu. Design and estimation of coded exposure point spread functions. IEEE Trans. Pattern Anal. Mach. Intell., 34, 2071-2077(2012).

[23] H.-G. Jeon, J.-Y. Lee, Y. Han, S. J. Kim, I. S. Kweon. Generating fluttering patterns with low autocorrelation for coded exposure imaging. Int. J. Comput. Vis., 123, 269-286(2017).

[24] J. Chai, H. Zeng, A. Li, E. W. T. Ngai. Deep learning in computer vision: a critical review of emerging techniques and application scenarios. Mach. Learn. Appl., 6, 100134(2021).

[25] S. Nah, T. H. Kim, K. M. Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 257-265(2017).

[26] X. Tao, H. Gao, X. Shen, J. Wang, J. Jia. Scale-recurrent network for deep image deblurring. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8174-8182(2018).

[27] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, L. Shao. Multi-stage progressive image restoration. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14816-14826(2021).

[28] S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, S.-J. Ko. Rethinking coarse-to-fine approach in single image deblurring. IEEE/CVF International Conference on Computer Vision (ICCV), 4621-4630(2021).

[29] X. Mao, Y. Liu, W. Shen, Q. Li, Y. Wang. Deep residual Fourier transformation for single image deblurring. arXiv(2021).

[30] K. Kim, S. Lee, S. Cho. MSSNet: multi-scale-stage network for single image deblurring. Computer Vision–ECCV, 13802, 524-539(2023).

[31] D. P. Kingma, M. Welling. Auto-encoding variational bayes. arXiv(2022).

[32] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, A. A. Bharath. Generative adversarial networks: an overview. IEEE Signal Process. Mag., 35, 53-65(2018).

[33] C. Hu, H. Huang, M. Chen, S. Yang, H. Chen. Video object detection from one single image through opto-electronic neural network. APL Photonics, 6, 046104(2021).

[34] Y. Liang, H. Huang, J. Li, X. Dong, M. Chen, S. Yang, H. Chen. Action recognition based on discrete cosine transform by optical pixel-wise encoding. APL Photonics, 7, 116101(2022).

[35] Z. Zhang, B. Zhang, X. Yuan, S. Zheng, X. Su, J. Suo, D. J. Brady, Q. Dai. From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth. PhotoniX, 3, 19(2022).

[36] M. Courbariaux, Y. Bengio, J.-P. David. BinaryConnect: training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems (NeurIPS), 28(2015).

[37] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, Y. Bengio. Binarized neural networks. Advances in Neural Information Processing Systems (NeurIPS), 29(2016).

[38] J. Cao, Y. Li, M. Sun, Y. Chen, D. Lischinski, D. Cohen-Or, B. Chen, C. Tu. DO-Conv: depthwise over-parameterized convolutional layer. IEEE Trans. Image Process., 31, 3726-3736(2022).

[39] K. He, X. Zhang, S. Ren, J. Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778(2016).

[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala. PyTorch: an imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 8024-8035(2019).

[41] D. P. Kingma, J. Ba. Adam: a method for stochastic optimization. arXiv(2017).

[42] I. Loshchilov, F. Hutter. SGDR: stochastic gradient descent with warm restarts. 5th International Conference on Learning Representations (ICLR)(2017).

[43] X. Chu, L. Chen, C. Chen, X. Lu. Improving image restoration by revisiting global information aggregation. Computer Vision–ECCV, 53-71(2022).

[44] G. Cui, X. Ye, J. Zhao, L. Zhu, Y. Chen, Y. Zhang. An effective coded exposure photography framework using optimal fluttering pattern generation. Opt. Lasers Eng., 139, 106489(2021).