• Advanced Photonics
  • Vol. 5, Issue 6, 066003 (2023)
Xin Tong1、2, Renjun Xu2, Pengfei Xu1, Zishuai Zeng1, Shuxi Liu1, and Daomu Zhao1、*
Author Affiliations
  • 1Zhejiang University, School of Physics, Zhejiang Province Key Laboratory of Quantum Technology and Device, Hangzhou, China
  • 2Zhejiang University, Center for Data Science, Hangzhou, China
  • show less
    DOI: 10.1117/1.AP.5.6.066003 Cite this Article Set citation alerts
    Xin Tong, Renjun Xu, Pengfei Xu, Zishuai Zeng, Shuxi Liu, Daomu Zhao. Harnessing the magic of light: spatial coherence instructed swin transformer for universal holographic imaging[J]. Advanced Photonics, 2023, 5(6): 066003 Copy Citation Text show less
    Principle and performance of TWC-Swin method. (a) LPR. SC modulation can adjust the SC by changing the distance D. Holographic modulation is used to load the phase hologram. The LPR generates two outputs, one for calculating SC and the other for network input. HWP, half-wave plate; PBS, polarized beam splitter; L, lens; RD, rotating diffuser; SLM, spatial light modulator; F, filter. D, distance between L1 and RD. (b) The detailed flow of the TWC-Swin method. The swin adapter can select the optimal model from the model space by obtaining SC. The color picture represents a case in progress. (c) Swin-model space and architecture of the swin model. The architecture of M1−M11 is the same; only the weights are different. The weights are obtained by network training at different distances. (d) The correspondence between SC and swin-model space. See Table S1 in the Supplementary Material for detailed data. (e) Inputs and outputs of the swin model with different SCs. (f) SSIM and PCC of swin-model outputs at different SCs. (g) Training and test data acquisition process. The training data did not contain any turbulence. (h) SSIM and PCC of swin-model outputs at different turbulent scenes.
    Fig. 1. Principle and performance of TWC-Swin method. (a) LPR. SC modulation can adjust the SC by changing the distance D. Holographic modulation is used to load the phase hologram. The LPR generates two outputs, one for calculating SC and the other for network input. HWP, half-wave plate; PBS, polarized beam splitter; L, lens; RD, rotating diffuser; SLM, spatial light modulator; F, filter. D, distance between L1 and RD. (b) The detailed flow of the TWC-Swin method. The swin adapter can select the optimal model from the model space by obtaining SC. The color picture represents a case in progress. (c) Swin-model space and architecture of the swin model. The architecture of M1M11 is the same; only the weights are different. The weights are obtained by network training at different distances. (d) The correspondence between SC and swin-model space. See Table S1 in the Supplementary Material for detailed data. (e) Inputs and outputs of the swin model with different SCs. (f) SSIM and PCC of swin-model outputs at different SCs. (g) Training and test data acquisition process. The training data did not contain any turbulence. (h) SSIM and PCC of swin-model outputs at different turbulent scenes.
    Qualitative analysis of our method’s performance at the different SCs. Input, raw image captured by CMOS1. Output, image processed by the network. (a)–(k) Different SCs: (a) D=f1, SC is 0.494; (b) D=1.1f1, SC is 0.475; (c) D=1.2f1, SC is 0.442; (d) D=1.3f1, SC is 0.419; (e) D=1.4f1, SC is 0.393; (f) D=1.5f1, SC is 0.368; (g) D=1.6f1, SC is 0.337; (h) D=1.7f1, SC is 0.311; (i) D=1.8f1, SC is 0.285; (j) D=1.9f1, SC is 0.25; and (k) D=2f1, SC is 0.245. D means the distance between L1 and RD in the LPR and f1 is the focal length of L1. Our method can achieve improved image quality under low SC (Video 1, MP4, 1.5 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s1]).
    Fig. 2. Qualitative analysis of our method’s performance at the different SCs. Input, raw image captured by CMOS1. Output, image processed by the network. (a)–(k) Different SCs: (a) D=f1, SC is 0.494; (b) D=1.1f1, SC is 0.475; (c) D=1.2f1, SC is 0.442; (d) D=1.3f1, SC is 0.419; (e) D=1.4f1, SC is 0.393; (f) D=1.5f1, SC is 0.368; (g) D=1.6f1, SC is 0.337; (h) D=1.7f1, SC is 0.311; (i) D=1.8f1, SC is 0.285; (j) D=1.9f1, SC is 0.25; and (k) D=2f1, SC is 0.245. D means the distance between L1 and RD in the LPR and f1 is the focal length of L1. Our method can achieve improved image quality under low SC (Video 1, MP4, 1.5 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s1]).
    Average results of the evaluation indices for each test data set. The coherence is 0.368. Results of other coherences are provided in Fig. S2 in the Supplementary Material. All evaluation indices demonstrate that our method possesses strong image restoration ability under low SC.
    Fig. 3. Average results of the evaluation indices for each test data set. The coherence is 0.368. Results of other coherences are provided in Fig. S2 in the Supplementary Material. All evaluation indices demonstrate that our method possesses strong image restoration ability under low SC.
    Qualitative analysis of our method’s performance across varying intensities of (a) oceanic and (b) atmospheric turbulence. The network trained with coherence as physical prior information can effectively overcome the impact of turbulence on imaging and improve image quality. (O1)–(O5) mean oceanic turbulence phase and (A1)–(A5) mean atmospheric turbulence phase. (O1) χt=10−9 K2/s, coherence is 0.491. (O2) χt=10−7 K2/s, coherence is 0.482. (O3) χt=2×10−7 K2/s, coherence is 0.447. (O4) χt=4×10−7 K2/s, coherence is 0.404. (O5) χt=10−6 K2/s, coherence is 0.373. (A1) Cn2=10−14 m3−α, coherence is 0.507. (A2) Cn2=1.5×10−13 m3−α, coherence is 0.459. (A3) Cn2=2.5×10−13 m3−α, coherence is 0.43. (A4) Cn2=3.5×10−13 m3−α, coherence is 0.403. (A5) Cn2=5×10−13 m3−α, coherence is 0.378. Other parameter settings of the turbulent power spectrum function can be found in Table S2 in the Supplementary Material (Video 2, MP4, 36.4 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s2]).
    Fig. 4. Qualitative analysis of our method’s performance across varying intensities of (a) oceanic and (b) atmospheric turbulence. The network trained with coherence as physical prior information can effectively overcome the impact of turbulence on imaging and improve image quality. (O1)–(O5) mean oceanic turbulence phase and (A1)–(A5) mean atmospheric turbulence phase. (O1) χt=109  K2/s, coherence is 0.491. (O2) χt=107  K2/s, coherence is 0.482. (O3) χt=2×107  K2/s, coherence is 0.447. (O4) χt=4×107  K2/s, coherence is 0.404. (O5) χt=106  K2/s, coherence is 0.373. (A1) Cn2=1014  m3α, coherence is 0.507. (A2) Cn2=1.5×1013  m3α, coherence is 0.459. (A3) Cn2=2.5×1013  m3α, coherence is 0.43. (A4) Cn2=3.5×1013  m3α, coherence is 0.403. (A5) Cn2=5×1013  m3α, coherence is 0.378. Other parameter settings of the turbulent power spectrum function can be found in Table S2 in the Supplementary Material (Video 2, MP4, 36.4 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s2]).
    Visualization of performance of different methods. The SSIM is shown in the bottom left corner. Our method presents the best performance, which is shown by smoother images with lower noise. (a) Sample selected with the WED data set and magnified insets of the red bounding region. (b) Sample selected with Flickr data set and magnified insets of the red bounding region. The pure swin model can be obtained by removing the postprocessing block of the swin model (Video 3, MP4, 0.6 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s3]).
    Fig. 5. Visualization of performance of different methods. The SSIM is shown in the bottom left corner. Our method presents the best performance, which is shown by smoother images with lower noise. (a) Sample selected with the WED data set and magnified insets of the red bounding region. (b) Sample selected with Flickr data set and magnified insets of the red bounding region. The pure swin model can be obtained by removing the postprocessing block of the swin model (Video 3, MP4, 0.6 MB [URL: https://doi.org/10.1117/1.AP.5.6.066003.s3]).
    Performance between different methods on various data sets with SC being 0.494. Our model outperforms other methods across various data sets and indices.
    Fig. 6. Performance between different methods on various data sets with SC being 0.494. Our model outperforms other methods across various data sets and indices.
    (a), (b) Performance comparison between different methods at various turbulent scenes. (A1) Cn2=10−14 m3−α, coherence is 0.506. (A2) Cn2=1.5×10−13 m3−α, coherence is 0.459. (O1) χt=10−9 K2/s, coherence is 0.491. (O2) χt=10−7 K2/s, coherence is 0.482. Note that all methods are trained with coherence as physical prior information and improve image quality under turbulence conditions. This demonstrates that incorporating appropriate physical prior information can help the network cope with multiscene tasks.
    Fig. 7. (a), (b) Performance comparison between different methods at various turbulent scenes. (A1) Cn2=1014  m3α, coherence is 0.506. (A2) Cn2=1.5×1013  m3α, coherence is 0.459. (O1) χt=109  K2/s, coherence is 0.491. (O2) χt=107  K2/s, coherence is 0.482. Note that all methods are trained with coherence as physical prior information and improve image quality under turbulence conditions. This demonstrates that incorporating appropriate physical prior information can help the network cope with multiscene tasks.
    SCSSIMPCC
    BSDCelebAFlickrWEDDIVBSDCelebAFlickrWEDDIV
    Input_f1, SC = 0.4940.58930.59430.42960.61550.46250.93680.95750.92100.91460.8753
    Output_f10.89840.89080.85230.90190.89400.98070.98930.98480.99300.9819
    Input_1.3f1, SC = 0.4190.57750.54150.39170.62450.41840.89530.93030.85880.91490.8043
    Output_1.3f10.91890.88420.86760.89970.89180.98430.99280.98800.99280.9827
    Input_1.5f1, SC = 0.3680.61780.53940.27770.56770.38920.89570.92110.83960.89610.8144
    Output_1.5f10.89060.85130.81710.85410.86220.96910.98810.97830.98690.9680
    Input_1.7f1, SC = 0.3110.60400.50170.31830.55100.41360.83030.90350.85110.85680.7979
    Output_1.7f10.86240.77910.74830.80130.80380.96440.97870.97020.97590.9583
    Input_2f1, SC = 0.2450.48810.44690.30730.52710.36430.80720.88170.75570.83260.7196
    Output_2f10.81460.75400.69620.77220.75720.94310.97130.95050.96310.9341
    Ground truth1111111111
    Table 1. Quantitative analysis of evaluation indices (SSIM and PCC) at different SCs and test samplesa. f1 is the focal length of L1. SC means spatial coherence of the light source.
    Oceanic turbulenceSSIMPCC
    BSDCelebAFlickrWEDDIVBSDCelebAFlickrWEDDIV
    Input (O1)0.53310.67730.68100.60160.70180.89780.94040.88760.90960.8718
    Output (O1)0.80880.79160.83680.80770.81720.93030.97070.93340.95600.9044
    Input (O2)0.50980.65660.66900.57160.53710.88550.93290.87860.89700.8494
    Output (O2)0.78230.76090.80150.78190.80050.92110.96110.92090.94480.8901
    Input (O3)0.49500.65380.65750.54550.52810.87640.93130.85850.89160.8371
    Output (O3)0.71910.71690.84340.73780.79840.88960.94130.88710.93440.8793
    Input (O4)0.47960.64080.64740.50340.50740.87740.92450.85760.86640.8130
    Output (O4)0.70600.69320.72870.67180.72170.88470.93790.88350.88920.8213
    Input (O5)0.45190.60410.62020.44460.49450.84560.90750.82870.82810.7631
    Output (O5)0.68990.67210.72250.62860.69580.89090.94150.88880.88390.8152
    Ground truth1111111111
    Table 2. Quantitative analysis of evaluation indices (SSIM and PCC) at different oceanic turbulence intensitiesa.
    Atmospheric turbulenceSSIMPCC
    BSDCelebAFlickrWEDDIVBSDCelebAFlickrWEDDIV
    Input (A1)0.57380.68210.69880.64950.63380.90140.94040.89290.91600.9766
    Output (A1)0.77980.77410.83370.81610.82310.93610.95640.92150.95740.9116
    Input (A2)0.53110.65130.67270.57430.57010.87970.92640.86760.88960.8279
    Output (A2)0.73120.69380.76990.69600.75810.89200.93530.89240.91410.8643
    Input (A3)0.50830.63830.67850.53480.57200.86880.92020.84930.87470.8081
    Output (A3)0.66150.67970.74270.63620.73690.88430.93920.87080.89190.8418
    Input (A4)0.49650.62640.66350.52020.55750.85900.91610.83640.86730.8040
    Output (A4)0.69150.67510.72870.63360.72730.87890.93080.87050.88550.8331
    Input (A5)0.49590.61530.65950.48400.54070.85240.90800.82630.84930.7862
    Output (A5)0.67610.68930.72010.61270.68020.87190.94650.88750.87490.8255
    Ground truth1111111111
    Table 3. Quantitative analysis of evaluation indices (SSIM and PCC) at different atmospheric turbulence intensitiesa.
    Xin Tong, Renjun Xu, Pengfei Xu, Zishuai Zeng, Shuxi Liu, Daomu Zhao. Harnessing the magic of light: spatial coherence instructed swin transformer for universal holographic imaging[J]. Advanced Photonics, 2023, 5(6): 066003
    Download Citation