• Infrared and Laser Engineering
  • Vol. 50, Issue 6, 20211033 (2021)
Sailing He1、2, Shuo Li1, Xiang Chen1, Zhanpeng Xu1、2, Qiuwan Bian1、2, Jing Luo1, and Longqiang Luo1
Author Affiliations
  • 1National Engineering Research Center for Optical Instruments, Zhejiang University Centre for Optical and Electromagnetic Research, Hangzhou 310058, China
  • 2Ningbo Research Institute, Zhejiang University, Ningbo 315100, China
  • show less
    DOI: 10.3788/IRLA20211033 Cite this Article
    Sailing He, Shuo Li, Xiang Chen, Zhanpeng Xu, Qiuwan Bian, Jing Luo, Longqiang Luo. Application of hyperspectral imager and lidar in marine biological detection[J]. Infrared and Laser Engineering, 2021, 50(6): 20211033 Copy Citation Text show less
    Schematic diagram of microscopic hyperspectral imager and the results of system calibration. (a) A photo of the prototype microscopic hyperspectral imager; (b) Schematic illustration of the optical elements: 1. Objective, 2. Imaging lens, 3. Slit, 4. Collimator lens, 5. Prism, 6. Grating, 7. Prism, 8. Tube lens, 9. CMOS; (c) Original spectral image of a calibration source; (d) Spectrum of the calibration source measured by our system. The spectral resolution is about 3 nm; (e) Calibration result between the wavelength and pixel index; (f) Spectrum of the calibration source measured by a commercial spectrometer; (g) Reconstructed hyperspectral image of the resolution test target. Resolvable lines in element 6 of group 6 (4.39 μm); (h) Prototype of the whole working system[27]
    Fig. 1. Schematic diagram of microscopic hyperspectral imager and the results of system calibration. (a) A photo of the prototype microscopic hyperspectral imager; (b) Schematic illustration of the optical elements: 1. Objective, 2. Imaging lens, 3. Slit, 4. Collimator lens, 5. Prism, 6. Grating, 7. Prism, 8. Tube lens, 9. CMOS; (c) Original spectral image of a calibration source; (d) Spectrum of the calibration source measured by our system. The spectral resolution is about 3 nm; (e) Calibration result between the wavelength and pixel index; (f) Spectrum of the calibration source measured by a commercial spectrometer; (g) Reconstructed hyperspectral image of the resolution test target. Resolvable lines in element 6 of group 6 (4.39 μm); (h) Prototype of the whole working system[27]
    (a) Schematic diagram of the shadow generation under lateral illumination condition; (b) Illustration of multi-mode detections. Reflection mode and fluorescence mode use epi-illumination, while transmission mode employs trans-illumination. Schematic diagram of each component: 1. Single mode fiber, 2. Fiber collimator, 3. Doublet lens, 4. Beam splitter, 5. Long pass filter, 6. Objective, 7. Sample and motion stage, 8. Infinity-corrected hyperspectral imaging system, consisting of the imaging module[27]
    Fig. 2. (a) Schematic diagram of the shadow generation under lateral illumination condition; (b) Illustration of multi-mode detections. Reflection mode and fluorescence mode use epi-illumination, while transmission mode employs trans-illumination. Schematic diagram of each component: 1. Single mode fiber, 2. Fiber collimator, 3. Doublet lens, 4. Beam splitter, 5. Long pass filter, 6. Objective, 7. Sample and motion stage, 8. Infinity-corrected hyperspectral imaging system, consisting of the imaging module[27]
    Experimental results the hyperspectral detection of a zebrafish under the transmission mode. (a) Optical microscopic image of a zebrafish underwater. Scale bar: 100 μm; (b) Hyperspectral image of a zebrafish reconstructed from our system. Scale bar: 100 μm; (c) Partial enlarged image of the zebrafish and three points of interest, including the transparent fin (marked as red dot), speckle (marked as blue star) and yolk sac (marked as yellow triangle); (d) Corresponding transmittance spectrum at the three points of interest[27]
    Fig. 3. Experimental results the hyperspectral detection of a zebrafish under the transmission mode. (a) Optical microscopic image of a zebrafish underwater. Scale bar: 100 μm; (b) Hyperspectral image of a zebrafish reconstructed from our system. Scale bar: 100 μm; (c) Partial enlarged image of the zebrafish and three points of interest, including the transparent fin (marked as red dot), speckle (marked as blue star) and yolk sac (marked as yellow triangle); (d) Corresponding transmittance spectrum at the three points of interest[27]
    [in Chinese]
    Fig. 4. [in Chinese]
    (a) Measured fluorescence spectra of five disaster-causing microalgae by the MMHI system; (b) Normalized spectra of (a); (c) Measured fluorescence spectra of five disaster-causing microalgae by a commercial spectrometer; (d) Normalized spectra of (c)[27]
    Fig. 4. (a) Measured fluorescence spectra of five disaster-causing microalgae by the MMHI system; (b) Normalized spectra of (a); (c) Measured fluorescence spectra of five disaster-causing microalgae by a commercial spectrometer; (d) Normalized spectra of (c)[27]
    Classification with the experimental data for five disaster-causing algae samples. (a) The relationship between PC1 and PC2; (b) The relationship among PC1, PC2 and PC3[27]
    Fig. 5. Classification with the experimental data for five disaster-causing algae samples. (a) The relationship between PC1 and PC2; (b) The relationship among PC1, PC2 and PC3[27]
    (a) The photo of our rotational hyperspectral scanner; (b) Schematic diagram and optical elements (1. imaging lens; 2. line slit; 3. collimating lens; 4. grating; 5. prism; 6. imaging lens); (c) Installed on a rotational mount[29]
    Fig. 6. (a) The photo of our rotational hyperspectral scanner; (b) Schematic diagram and optical elements (1. imaging lens; 2. line slit; 3. collimating lens; 4. grating; 5. prism; 6. imaging lens); (c) Installed on a rotational mount[29]
    (a) Reconstructed image by short scan line; (b) Reconstructed image by long scan line; (c) Correlation coefficient chart; (d) Optimum reconstructed image[29]
    Fig. 7. (a) Reconstructed image by short scan line; (b) Reconstructed image by long scan line; (c) Correlation coefficient chart; (d) Optimum reconstructed image[29]
    [in Chinese]
    Fig. 7. [in Chinese]
    (a) Photo of a building; (b) Image reconstructed by rotational hyperspectral scanner;(c) Effect of elimination of middle missing hole[29]
    Fig. 8. (a) Photo of a building; (b) Image reconstructed by rotational hyperspectral scanner;(c) Effect of elimination of middle missing hole[29]
    [in Chinese]
    Fig. 9. [in Chinese]
    (a) Full spectral image; (b) CIE-1931 RGB stimulation coefficient; (c) Color restoration; (d) Color checker[29]
    Fig. 9. (a) Full spectral image; (b) CIE-1931 RGB stimulation coefficient; (c) Color restoration; (d) Color checker[29]
    (a) Photo of fruits; (b) Hyperspectral image; (c) Spectral plot[29]
    Fig. 10. (a) Photo of fruits; (b) Hyperspectral image; (c) Spectral plot[29]
    [in Chinese]
    Fig. 10. [in Chinese]
    Spectral image at different wavelength bands[29]
    Fig. 11. Spectral image at different wavelength bands[29]
    Hyperspectral images of three species of microalgae. (a) Phaeocystis; (b) Chlam-mydomonas; (c) Chaetoceros. Inserted (blue) images show their corresponding optical microscopic images. Scale bar: 50 μm[31]
    Fig. 12. Hyperspectral images of three species of microalgae. (a) Phaeocystis; (b) Chlam-mydomonas; (c) Chaetoceros. Inserted (blue) images show their corresponding optical microscopic images. Scale bar: 50 μm[31]
    Verification of absorption characteristics of three microalgae. (a) Original transmission spectra of the LED light source and three microalgae; (b) Normalized spectra of (a); (c) Transmittance of the microalgae relative to the LED light source; (d) Absorbance of the microalgae measured by a commercial spectrophotometer[31]
    Fig. 13. Verification of absorption characteristics of three microalgae. (a) Original transmission spectra of the LED light source and three microalgae; (b) Normalized spectra of (a); (c) Transmittance of the microalgae relative to the LED light source; (d) Absorbance of the microalgae measured by a commercial spectrophotometer[31]
    [in Chinese]
    Fig. 14. [in Chinese]
    (a) Scatter plot of the scores of PC2 versus PC1 of three microalgae along with the linear SVM classifiers. The entire two-dimensional space is divided into three regions as labeled; (b) Confusion matrix of all transmission spectra for 180 samples (including 60 chaetoceros, 60 chlamydomonas and 60 phaeocystis); (c) The ROC curves corresponding to the SVM classifiers in (a)[31]
    Fig. 14. (a) Scatter plot of the scores of PC2 versus PC1 of three microalgae along with the linear SVM classifiers. The entire two-dimensional space is divided into three regions as labeled; (b) Confusion matrix of all transmission spectra for 180 samples (including 60 chaetoceros, 60 chlamydomonas and 60 phaeocystis); (c) The ROC curves corresponding to the SVM classifiers in (a)[31]
    Demonstration for the species identification from a group of mixed microalgae. (a) Hyperspectral image of the mixed microalgae at the broad visible bands; (b) Hyperspectral image of the mixed microalgae at the single spectral band (680 nm); (c) The result of species identification on (a) after adding the image mask, the blue and red regions represent the chlamydomonas and phaeocystis, respectively; (d) The template image mask, generated by threshold segmentation on (b)[31]
    Fig. 15. Demonstration for the species identification from a group of mixed microalgae. (a) Hyperspectral image of the mixed microalgae at the broad visible bands; (b) Hyperspectral image of the mixed microalgae at the single spectral band (680 nm); (c) The result of species identification on (a) after adding the image mask, the blue and red regions represent the chlamydomonas and phaeocystis, respectively; (d) The template image mask, generated by threshold segmentation on (b)[31]
    (a) Growth curve of phaeocystis over 25 days. 1: lag phase, 2: exponential growth phase, 3: stable phase, 4: decline phase; (b) Normalized transmission spectra of phaeocystis; (c) Predicted results of the growth stage by the training set; (d) Predicted results of the growth stage by the test set[31]
    Fig. 16. (a) Growth curve of phaeocystis over 25 days. 1: lag phase, 2: exponential growth phase, 3: stable phase, 4: decline phase; (b) Normalized transmission spectra of phaeocystis; (c) Predicted results of the growth stage by the training set; (d) Predicted results of the growth stage by the test set[31]
    [in Chinese]
    Fig. 16. [in Chinese]
    Schematic diagram of Scheimpflug principle. O is the origin of coordinate system; O' is the center of the lens; O'' is the coordinate origin of the image sensor; is the included angle between image plane and lens plane; is the included angle between object plane and lens plane; is the length between the center of the image plane and the center of the lens; d is the length from the center of the lens to the object plane; is is the pixel position of the image point on the image sensor; A and B are object points in the object plane, A' and B' are image points corresponding to A and B沙姆原理示意图。O是坐标系的原点;O'是透镜的中心;O''是图像传感器的坐标原点;是像平面与透镜平面之间的夹角;是物平面与透镜平面之间的夹角;是像面中心与透镜中心的距离;是透镜中心到物平面的距离;是图像传感器上像点的像素位置;A、B为物平面上的物点,A'、B'为A、B对应的像点
    Fig. 17. Schematic diagram of Scheimpflug principle. O is the origin of coordinate system; O' is the center of the lens; O'' is the coordinate origin of the image sensor; is the included angle between image plane and lens plane; is the included angle between object plane and lens plane; is the length between the center of the image plane and the center of the lens; d is the length from the center of the lens to the object plane; is is the pixel position of the image point on the image sensor; A and B are object points in the object plane, A' and B' are image points corresponding to A and B沙姆原理示意图。O是坐标系的原点;O'是透镜的中心;O''是图像传感器的坐标原点; 是像平面与透镜平面之间的夹角; 是物平面与透镜平面之间的夹角; 是像面中心与透镜中心的距离; 是透镜中心到物平面的距离; 是图像传感器上像点的像素位置;AB为物平面上的物点,A'、B'为AB对应的像点
    Optical layout of distance correction. O is the origin of coordinate system; O' is the center of the lens; is the position of each pixel of the image sensor on the image plane; A0 is the position of the object without considering the refraction of the air-water interface; A1 is the actual position of the object; A1' is the image point corresponding to A1; is the angle between the incident light and the vertical direction; is the angle between the refracted light and the vertical direction; z0 is the distance between the origin of the coordinates (O) and the air-water interface; z is the uncorrected distance (OA0); z1 is the actual distance (OA1) after correction距离矫正示意图。O是坐标系的原点;O'是透镜的中心;是像平面上图像传感器每个像素的位置;A0是未考虑空气-水界面折射时物体的位置;A1是物体实际的位置;A1'是A1对应的像点;是入射光束与垂直方向的夹角;是折射光束与垂直方向的夹角;z0是坐标原点(O)与空气-水界面之间的距离;z是未矫正的距离(OA0);z1是矫正后实际的距离(OA1)
    Fig. 18. Optical layout of distance correction. O is the origin of coordinate system; O' is the center of the lens; is the position of each pixel of the image sensor on the image plane; A0 is the position of the object without considering the refraction of the air-water interface; A1 is the actual position of the object; A1' is the image point corresponding to A1; is the angle between the incident light and the vertical direction; is the angle between the refracted light and the vertical direction; z0 is the distance between the origin of the coordinates (O) and the air-water interface; z is the uncorrected distance (OA0); z1 is the actual distance (OA1) after correction 距离矫正示意图。O是坐标系的原点;O'是透镜的中心; 是像平面上图像传感器每个像素的位置;A0是未考虑空气-水界面折射时物体的位置;A1是物体实际的位置;A1'是A1对应的像点; 是入射光束与垂直方向的夹角; 是折射光束与垂直方向的夹角;z0是坐标原点(O)与空气-水界面之间的距离;z是未矫正的距离(OA0);z1是矫正后实际的距离(OA1
    (a) A prototype of inelastic hyperspectral Scheimpflug lidar system; (b) Inelastic hyperspectral Scheimpflug lidar imaging schematic: L1 and L2 are collimated lenses, and OF is a long-pass optical filter. P1 and P2 are two symmetrical wedge prisms, and G is a transmission grating with 300 grooves per mm
    Fig. 19. (a) A prototype of inelastic hyperspectral Scheimpflug lidar system; (b) Inelastic hyperspectral Scheimpflug lidar imaging schematic: L1 and L2 are collimated lenses, and OF is a long-pass optical filter. P1 and P2 are two symmetrical wedge prisms, and G is a transmission grating with 300 grooves per mm
    (a) Distance calibration result; (b) Spectral calibration result
    Fig. 20. (a) Distance calibration result; (b) Spectral calibration result
    (a) A sample of blue Cassiopea andromedah; (b) A sample of brown Cassiopea andromeda; (c) A sample of Mastigias papua
    Fig. 21. (a) A sample of blue Cassiopea andromedah; (b) A sample of brown Cassiopea andromeda; (c) A sample of Mastigias papua
    Fluorescence hyperspectra of three kinds of jellyfish measured by hyperspectral Scheimpflug lidar: (a) The wavelength of excitation light is 446 nm; (b) The wavelength of excitation light is 488 nm
    Fig. 22. Fluorescence hyperspectra of three kinds of jellyfish measured by hyperspectral Scheimpflug lidar: (a) The wavelength of excitation light is 446 nm; (b) The wavelength of excitation light is 488 nm
    (a) A photograph of phaeocystis globosa showed that the diameter of large cysts could reach 15-16 mm; (b) Normalized fluorescence spectra of phaeocystis globosa cysts measured in the onshore tank experiment; (c) A photograph of on-site measurement in the nearshore fishing ground; (d) Normalized fluorescence spectra of phaeocystis globosa cysts measured in the nearshore fishing ground
    Fig. 23. (a) A photograph of phaeocystis globosa showed that the diameter of large cysts could reach 15-16 mm; (b) Normalized fluorescence spectra of phaeocystis globosa cysts measured in the onshore tank experiment; (c) A photograph of on-site measurement in the nearshore fishing ground; (d) Normalized fluorescence spectra of phaeocystis globosa cysts measured in the nearshore fishing ground
    (a) Aequorea; (b) Diphyidae
    Fig. 24. (a) Aequorea; (b) Diphyidae
    (a) Normalized fluorescence spectra of aequorea; (b) Normalized fluorescence spectra of diphyidae
    Fig. 25. (a) Normalized fluorescence spectra of aequorea; (b) Normalized fluorescence spectra of diphyidae
    (a) HSDA system, including a front optical module, hyperspectral imager and structured light stereovision; (b) Schematic diagram of HSDA system; (c) Physical map of HSDA system[32]
    Fig. 26. (a) HSDA system, including a front optical module, hyperspectral imager and structured light stereovision; (b) Schematic diagram of HSDA system; (c) Physical map of HSDA system[32]
    Flow chart of system.(a) Original 4D model; (b) Spectrum curves of point A, B, C and D in the origin model; (c) 4D model with point cloud segmentation based on spectral information; (d) Monochrome images at different wavelengths (450-750 nm) for different 3D perspectives[32]
    Fig. 27. Flow chart of system.(a) Original 4D model; (b) Spectrum curves of point A, B, C and D in the origin model; (c) 4D model with point cloud segmentation based on spectral information; (d) Monochrome images at different wavelengths (450-750 nm) for different 3D perspectives[32]
    (a) The green plant; (b) The hyperspectral cube of the measured green plant; (c)3D point cloud model of the green plant; (d) The 4D data of the green plant observed at different angles and at different wavelengths (450-675 nm); (e) Enlargement of the leaf blade margin; (f) The distribution of the point cloud of blue rectangular region in the leaf blade; (g) The surface patch of the point cloud[32]
    Fig. 28. (a) The green plant; (b) The hyperspectral cube of the measured green plant; (c)3D point cloud model of the green plant; (d) The 4D data of the green plant observed at different angles and at different wavelengths (450-675 nm); (e) Enlargement of the leaf blade margin; (f) The distribution of the point cloud of blue rectangular region in the leaf blade; (g) The surface patch of the point cloud[32]
    [in Chinese]
    Fig. 29. [in Chinese]
    3D point cloud of the (a) green plant and (b) plastic plant displayed using RGB colors; (c) Intensity spectrum of 3D points A and point B; (d) Normalized reflectance spectrum of 3D points A and B[32]
    Fig. 29. 3D point cloud of the (a) green plant and (b) plastic plant displayed using RGB colors; (c) Intensity spectrum of 3D points A and point B; (d) Normalized reflectance spectrum of 3D points A and B[32]
    Specifications of the HSDA systemCharacteristic parameters
    Spectral range/nm400-800
    Spectral resolution/nm<3
    Depth resolution/mm0.0275
    Plane fit standard deviation/mm0.0269
    Num of 4D points<800×800
    Acquisition time/s<80
    FOV/(°)30
    Measurement field/mm296×64
    Working distance/mm300-600
    Table 1. Four dimensional hyperspectral detection system parameters
    Sailing He, Shuo Li, Xiang Chen, Zhanpeng Xu, Qiuwan Bian, Jing Luo, Longqiang Luo. Application of hyperspectral imager and lidar in marine biological detection[J]. Infrared and Laser Engineering, 2021, 50(6): 20211033
    Download Citation