• PhotoniX
  • Vol. 3, Issue 1, 19 (2022)
Zhihong Zhang1、†, Bo Zhang1、†, Xin Yuan2、3、†, Siming Zheng4、5, Xiongfei Su2, Jinli Suo1、*, David J. Brady6, and Qionghai Dai1、**
Author Affiliations
  • 1Department of Automation, Tsinghua University, 100084 Beijing, China
  • 2Research Center for Industries of the Future and School of Engineering, Westlake University, 310024 Hangzhou, China
  • 3Key Laboratory of 3D Micro/Nano Fabrication and Characterization of Zhejiang Province, Westlake University, 310024 Hangzhou, China
  • 4Computer Network Information Center, Chinese Academy of Sciences, 100190 Beijing, China
  • 5University of Chinese Academy of Sciences, 100049, Beijing, China
  • 6Wyant College of Optical Sciences, University of Arizona, AZ 85721 Tucson, USA
  • show less
    DOI: 10.1186/s43074-022-00065-1 Cite this Article
    Zhihong Zhang, Bo Zhang, Xin Yuan, Siming Zheng, Xiongfei Su, Jinli Suo, David J. Brady, Qionghai Dai. From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth[J]. PhotoniX, 2022, 3(1): 19 Copy Citation Text show less

    Abstract

    High-throughput imaging is highly desirable in intelligent analysis of computer vision tasks. In conventional design, throughput is limited by the separation between physical image capture and digital post processing. Computational imaging increases throughput by mixing analog and digital processing through the image capture pipeline. Yet, recent advances of computational imaging focus on the “compressive sampling”, this precludes the wide applications in practical tasks. This paper presents a systematic analysis of the next step for computational imaging built on snapshot compressive imaging (SCI) and semantic computer vision (SCV) tasks, which have independently emerged over the past decade as basic computational imaging platforms. SCI is a physical layer process that maximizes information capacity per sample while minimizing system size, power and cost. SCV is an abstraction layer process that analyzes image data as objects and features, rather than simple pixel maps. In current practice, SCI and SCV are independent and sequential. This concatenated pipeline results in the following problems: i) a large amount of resources are spent on task-irrelevant computation and transmission, ii) the sampling and design efficiency of SCI is attenuated, and iii) the final performance of SCV is limited by the reconstruction errors of SCI. Bearing these concerns in mind, this paper takes one step further aiming to bridge the gap between SCI and SCV to take full advantage of both approaches. After reviewing the current status of SCI, we propose a novel joint framework by conducting SCV on raw measurements captured by SCI to select the region of interest, and then perform reconstruction on these regions to speed up processing time. We use our recently built SCI prototype to verify the framework. Preliminary results are presented and the prospects for a joint SCI and SCV regime are discussed. By conducting computer vision tasks in the compressed domain, we envision that a new era of snapshot compressive imaging with limited end-to-end bandwidth is coming.
    Zhihong Zhang, Bo Zhang, Xin Yuan, Siming Zheng, Xiongfei Su, Jinli Suo, David J. Brady, Qionghai Dai. From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth[J]. PhotoniX, 2022, 3(1): 19
    Download Citation