Detecting underwater acoustic sources from the air can allow aircraft and surface vessels to monitor the underwater acoustic environment and can be potentially used in many commercial and military applications. This kind of cross-medium detection scheme smartly utilizes the low transmission loss features of light in air and sound in water, respectively. Therefore, it has attracted the attention of many researchers around the world. As early as 2004, researchers from the Naval Undersea Warfare Center of America took the lead in using a laser Doppler vibrometer (LDV) to measure the vibration of the water surface modulated by the underwater acoustic signal. To solve the data dropout problem caused by the hydrodynamic water surface, a surface normal glint tracker was employed^[1]. The system was also successfully used in cross water–air surface uplink communication in 2006^[2]. In 2015, researchers from the Naval Air Warfare Center of America utilized an LDV to detect the vibrations of the water’s surface caused by an underwater speaker, and audio signals ranging from 50 Hz to 5 kHz were successfully detected^[3]. In 2007, a scanning LDV was used to analyze underwater acoustic wavefields, and the wavefronts scattered by circular cross-sectional obstruction were correctly established^[4]. In 2020, researchers from Donghua University, China, exploited a five-channel fiber-based LDV for underwater acoustic field measurement. The system performance was tested in Qiandao Lake and the signal probing probability of the whole sensing system was up to 59.77%^[5].

Except for LDV, many other cross air–water detecting methods were reported. In 2010, Farrant introduced the concept of optical sonar^[6], which used a ${\mathrm{CO}}_2$ laser to illuminate water and generate acoustic waves as a probe signal in water. The acoustic wave was reflected by underwater objects and transmitted to the water’s surface to modulate the air–water interface, which was probed by a 532 nm laser, and the location and orientation of the underwater objects can be deduced. The optical sonar scheme is followed by many researchers. They usually used the water surface acoustic wave (WSAW) originated from hydroacoustic sources as a signal carrier^[7–11]. In 2015, researchers from the National Oceanic and Atmospheric Administration (NOAA) of America used the relationship between sound pressure and bubble void to deduce the underwater acoustic signal frequency through the return lidar signals from a collection of insonified bubbles, which were modulated by the underwater acoustic signal^[12].

Except for optical detection methods, millimeter-wave radar and airborne sonar systems were also used to detect underwater acoustic sources. In 2018, researchers from Massachusetts Institute of Technology, America, showed that water surface modulated by acoustic waves can be detected with a millimeter-wave frequency modulated carrier wave (FMCW) radar. The distance between the radar and the water surface is 30 cm. Standard cross water–air interface communication with bit rates up to 400 bps was achieved, and the system could operate correctly in the presence of surface waves with peak-to-peak amplitudes of up to 16 cm^[13]. In 2020, researchers from Stanford University, America, reported an airborne sonar system for underwater remote sensing and imaging. A 1070 nm ytterbium laser with a peak power of kilowatts was used to generate detecting acoustic waves and a specially designed capacitive micromachined ultrasonic transducer was used to detect deformation of the water surface at a 10 cm working distance. Underwater objects with a depth of 13 cm were correctly detected^[14].

From the existing reports, we can see that the LDV method is usually selected for cross-medium audio sensing because of its high sensitivity and technology maturity. However, LDV is a single-point detecting method, which has a small signal receive angle and faces the return signal dropout problem in hydrodynamic situations. The millimeter-wave radar and airborne sonar methods are robust for severe weather conditions, but come with the shortcomings of low sensing resolution originated from their working mechanism. The amplitudes of water surface acoustic waves caused by underwater acoustic sources usually lie in tens of nanometer levels^[8]. A method with high sensing resolution and a wide viewing angle is needed in practical applications.

Except for the LDV method, other optical vibration detection methods, including laser self-mixing^[15], holographic vibration measurement^[16], photography^[17,18], single-pixel imaging^[19], and laser speckle imaging vibration measurement^[20], have also been developed in recent years. Among these methods, the laser speckle imaging vibration measurement method has the advantages of a simple setup and flexible system configuration and has been investigated by many researchers. Correspondingly, many measuring schemes are presented, including area array camera imaging^[21–24], photodiode flux detection^[25,26], linear array camera imaging^[27–30], and rolling-shutter camera imaging^[18,31–35]. Among the existing schemes, the linear array camera speckle imaging scheme is selected in this paper for its advantages of wide sensing frequency range, high sensitivity, large viewing angle, fast processing speed, low data storage, and transmission requirement.

In this paper, a cross air–water interface hydroacoustic signal sensing system based on linear array camera imaging is constructed. Phase correlation and multichannel fusion algorithms in the phase domain are introduced to extract and enhance the vibration signals. Underwater acoustic signals with a frequency range from 200 Hz to 20 kHz are successfully detected with a relative frequency error of less than 0.5%. Several common artificial and natural underwater acoustic signals are correctly detected. To the authors’ knowledge, this is the first report of cross air–water interface underwater acoustic signal detection based on laser speckle imaging. The arrangement of this paper is as follows. The experimental setup and signal extraction algorithm are introduced in Sec. 2. The experimental results and analysis are presented in Sec. 3, and a brief conclusion is given in Sec. 4.

The experimental setup is shown in Fig. 1. The setup can be divided into three modules, including the detection module, the signal generation module, and the amplitude calibration module, respectively. A beam of 532 nm laser (JDSU CDPS532 M 50 mW) is reflected by a mirror and approximately vertically illuminated at the water surface of a glass tank (length 60 cm, width 45 cm, height 45 cm). An underwater speaker (Chinese kHz Electronics Technology LTD, model UWS-015A) is placed on the bottom of the tank, which is filled with drinking water to a height of 40 cm; the distance from the speaker’s upper surface to the water surface is about 30 cm. The speaker is driven by an audio amplifier (Yamaha RX-V490), and the signal source is either a signal generator or a computer. The audio signals of the speaker are transmitted to the air–water interface and generate water surface acoustic waves, which modulate the illuminated laser beam. The modulated laser beam is diffusely reflected by the air–water interface to the air and then collected by an imaging system. The imaging system is composed of an imaging lens (Nikon Sigma Zoom Master: 35–70 mm) and a linear array CMOS camera (Basler raL2048-48gm). The camera is controlled by the Pylon Viewer software (Basler), and the recorded speckles are stored in the computers in .bmp format. The maximum line rate of the camera is 51,000 lines per second and can be configured according to the requirements of the software. To be specially noted: the line rate is configured as 16,000 lines per second and each frame contains 8000 lines (corresponding to an atemporal period of 0.5 s). In the experiment, the incident probe laser beam should be avoided from incident upright on the underwater speaker and the transmitted beamlet should be reflected by another mirror in the other direction and forbidden to enter the image system. To increase the received light flux, a piece of 10 µm thickness aluminum foil is placed on the water surface in the experiment. A typical speckle pattern is shown as the inset in Fig. 1.

To record the absolute sound level, a commercially available hydrophone (CSSC 715th Institute of China: SAH2014-11) is used. The receiving sensitivity of the hydrophone is $-175.7\mathrm{dB}\mathrm{re}\mathrm{\mu Pa}$ at 10 kHz. The sensor head of the hydrophone is immersed below the water surface about 10 cm, and the vertical and horizontal distances between the hydrophone and the underwater speaker center are 15 and 9 cm, respectively. The output of the hydrophone is digitized by a data acquisition card and is stored in a computer. The sound pressure level (SPL) at the water surface can be estimated as the SPL at the hydrophone sensor head and can be calculated through the output $V_{\mathrm{rms}}$ of the hydrophone using Eq. (1), $$\mathrm{SPL}=20\log (V_{\mathrm{rms}})-\mathrm{RVS}\mathrm{dB}\mathrm{re}\mathrm{\mu Pa},$$(1)where RVS is the sensitivity of the hydrophone at the respective frequency.

The signal extraction algorithm is a modified version of the phase correlation algorithm that is reported in Ref. [29]. The phase correlation algorithm is considered a high-performance image alignment method^[36]. In this paper, phase correlation is combined with multichannel fusion algorithms to enhance the performance of the vibration signal extraction. The algorithms are briefly introduced in the following section.

As the speckle pattern does not change quickly, the movement of the speckle pattern can be considered a rigid body movement. The images recorded in the $i$th row $I_i(x)$ and the ($i+1$)th row $I_{i+1}(x)$ can be related with a spatial shift of $\mathrm{\Delta }x$, $$I_i(x)=I_{i+1}(x-\mathrm{\Delta }x).$$(2)

Assume that $F_i(u)$ and $F_{i+1}(u)$ are the Fourier transforms of $I_i(x)$ and $I_{i+1}(x)$, respectively; their cross-correlation can be written as $$\begin{gathered} {\mathrm{\Gamma }}_{i,i+1}(u)=F_i{(u)}^{*}{F}_{i+1}(u) \\ =A_i(u)A_{i+1}(u)\exp [i({\mathrm{\Phi }}_{i+1}-{\mathrm{\Phi }}_i)], \end{gathered}$$(3)where $u$ is the spatial coordinate in the frequency domain, $A_i$ is the real part of $F_i(u)$, and ${\mathrm{\Phi }}_i$ represents the phase of $F_i(u)$. Accordingly, the following relationship is fulfilled: $$\mathrm{\Delta }\mathrm{\Phi }={\mathrm{\Phi }}_{i+1}-{\mathrm{\Phi }}_i=p\mathrm{\Delta }x,$$(4)where $p$ is a constant coefficient. As different rows represent different sampling instants, $\mathrm{\Delta }\mathrm{\Phi }$ represents the variation of phase with time. After normalization, we have $$S(t)=\frac{F_i*F_{i+1}}{|F_i*||F_{i+1}|}=\frac{F_i*F_{i+1}}{|F_i*F_{i+1}|}=\exp (ip\mathrm{\Delta }x).$$(5)

Thus, $S(t)$ can represent the object vibration. As the Fourier transform $F_i(u)$ of a row image $I_i(x)$ is a row datum, the normalized correlation $S(t)$ is also a row datum, which is shown in Fig. 2. Therefore, different column groups can be considered as different sensing channels, and multichannel fusion can be used to enhance the signal reconstruction performance. In this paper, the phase-error-based filtering (PBF) method is selected as the multichannel data fusion algorithm^[37].

Two columns, $s_{\tau }(t)$ and $s_k(t)$, in $S(t)$ are selected and segmented into M frames (frame length 64 ms) using a Hanning window; then they are Fourier transformed as $S_{\tau }(\omega )$ and $S_k(\omega )$. The phase variation between $S_{\tau }(\omega )$ and $S_k(\omega )$ at the $m$th frame, ${\theta }_{\tau ,k,m}$, can be calculated as $${\theta }_{\tau ,k,m}(\omega )=X_{\tau ,m}(\omega )-X_{k,m}(\omega ),$$(6)where $X_{\tau ,m}(\omega )$ and $X_{k,m}(\omega )$ are the $m$th frame phases of $S_{\tau }(\omega )$ and $S_k(\omega )$, respectively.

The signal decay coefficient of individual channel $\tau$ can be calculated as $${\psi }_{\tau ,m}(\omega )={[{\mathrm{\Pi }}_{k=1,k\ne \tau }^C{\eta }_{\tau ,k,m}(\omega )]}^{\frac{1}{r}},$$(7)where $C$ is the total number of input channels, ${\eta }_{\tau ,k,m}(\omega )=1/[1+\gamma {\theta }_{\tau ,k,m}^2(\omega )]$ is the decaying function, and $r$ is the weighting factor, which is set to the total frames $M$. $\gamma$ is a constant and is empirically set to 5, which gives good performance in wide signal-to-noise ratio (SNR) ranges. The fusion of these filtered signals in the frequency domain can improve the demodulated signal, $$S_m^{\prime }(\omega )=\sum _{\tau =1}^c{\psi }_{\tau ,m}(\omega )S_{\tau ,m}(\omega ).$$(8)

The inverse Fourier transform of $S_m^{\prime }(\omega )$ to the temporal domain can obtain the enhanced signal $s_m^{\prime }(t)$. The final enhanced extracted signal $s^{\prime }(t)=\sum _{m=1}^M{s}_m^{\prime }(t)$. The data processing flow is introduced in Fig. 2. The implementation of these algorithms is described in the following: 1) Transform image to data matrix by MATLAB function imread; 2) set pixels with gray value smaller than 6 as 0 to filter noise; 3) fast Fourier transform (FFT) on rows and calculate the phase difference matrix of adjacent rows according to Eq. (5); 4) set the Hanning window frame length at 64 ms, step length 1 ms, and $\gamma =5$; 5) FFT for selected 10 channels; 6) powder bed fusion (PBF) according to Eqs. (6), (7), and (8); 7) IFFT on fused signal to get $s_m^{\prime }(t)$; 8) align and add all frame signals to get $s^{\prime }(t)$.

To fulfill the Nyquist sampling criteria, the line rate of camera is set to 16,000 lines/s for 200–8000 Hz signals. For 9–16 kHz signals, the line rate is set to 32,000 lines/s. For 17–20 kHz signals, the line rate is set to 44,000 lines/s. Except for the temporal sampling criteria, the spatial sampling criteria, that is, the average speckle size, should be bigger than 2 pixels and also must be fulfilled. This condition can be fulfilled through the adjustment of imaging parameters.

In the experiment, the path length from the laser source to the water surface is 1.44 m. The distance from the image sensor to the water surface is about 1.90 m. The imaging lens is defocused and focused at 0.70 m. The focus length of the image lens is adjusted to 50 mm and the aperture number is F/5.6. The beam waist radius of the 532 nm laser is about 0.3 mm. The laser spot size D illuminated on the object to be detected can be calculated according to the propagation of the laser beam and is 1.732 mm. Accordingly, the average object speckle size on the focusing plane is $\lambda L_1/D$ and is 0.368 mm ($\lambda$ is the laser wavelength and $L_1$ is the distance between the water surface and lens focus point). Consequently, the subjective speckle size can be calculated as $\lambda L_1\times f/(L_2\times D)$ and is about 26.3 µm ($f$ is the focus length of the imaging lens, and $L_2$ is the distance between the lens and the lens focus point). The pixel size of the camera is 7.0 µm. In all directions, the average speckle image occupies 3.8 pixels, thus meeting the requirement of the spatial sampling law.

First, the vibration detection performance of single-frequency sinusoid signals is tested. The experimental setup is the same as that shown in Fig. 1, and the experimental parameters are as introduced in Sec. 2. According to the algorithm in Sec. 2, the frequency of the source audio signal can be reconstructed. Typical experimental results are shown in Fig. 3. The data in Fig. 3 are normalized.

From Fig. 3, we can see that every signal is correctly reconstructed, and the frequency relative error is smaller than 0.2%. There are evident harmonics in Fig. 3(a); the authors think the main reason may be that the speaker plays the signal with harmonics. The harmonics of the 4, 14, and 20 kHz signals are filtered out in the processing procedure. Typical signal reconstruction results are summarized in Table 1, and the frequency relative errors are all smaller than 0.3%. The frequency relative errors before and after the multichannel fusion are compared; the results show no obvious improvement.

The sensibility of the vibration detecting system is verified with a commercial LDV vibrometer (Polytec OFV-3001 OFV-303). With a 300 Hz underwater audio signal, the minimum measurable amplitude is about 69 nm. While at 1 kHz, the minimum measurable amplitude is about 5 nm. In the latter case, the vibration waveform is not stable and can only be roughly estimated.

The absolute SPLs of different frequencies are measured with the hydrophone and are listed in Table 1. The values range from 153.9 to 180.6 dB re µPa. The peak SPL of the 10 kHz in-water acoustic signal was calculated to be 177.7 dB re µPa at the water surface from the reference hydrophone output voltage using Eq. (1) with $V_{\mathrm{rms}}=1.26\mathrm{V}$. The measured SPL is consistent with the results of other researchers^[1]. It should be noted that the SPLs listed in Table 1 are not the minimum detectable SPLs of the sensing system introduced in this paper.

In this section, the signal detection performance for burst audio signals of underwater marine animals and artificial vehicles is evaluated. First, a humpback whale audio signal is selected as a signal source to drive the underwater speaker. The typical waveforms and spectrograms of the original signal and the reconstructed signal are shown in Fig. 4, where PC means phase correlation and PC_PBF means phase correlation combined with phase-error-based filtering. It can be seen from the spectrograms that the characteristic frequencies are reconstructed appropriately. The SNR of the reconstructed signals is evaluated with the segmental SNR (SegSNR) algorithm^[38] and the average SegSNR of 20 reconstructed signals is $(-3.89\pm 0.32)\mathrm{dB}$. The intelligibility of the reconstructed signals is evaluated with the normalized subband envelope correlation (NSEC) algorithm^[39]. The score of NSEC lies between 0 and 1, and 1 means the intelligibility is as good as the original audio signal. Usually, an NSEC better than 0.30 means the audio can be intelligible. The average NSEC score of 20 reconstructed signals is $0.48\pm 0.03$. The log-likelihood ratio (LLR) algorithm is used to estimate the likelihood between the original and recovered signals^[35]. The LLR score ranges from 0 to 2, and 0 means the best likelihood. The average LLR score of 20 reconstructed signals is $1.71\pm 0.11$. The results show that the SegSNR and the intelligibility performance are good.

To further enhance the recovered signals, the PBF multichannel fusion algorithm is employed. Ten channels with the highest scales are selected and fused together using the algorithm introduced in Sec. 2. The typical fusion results of the humpback whale audio signal are shown in Fig. 4(c), and the evaluation results are summarized in Table 2. Compared with the PC method, the SegSNR, NSEC score, and LLR score of the PC_PBF method for humpback whale signals are increased by 1.61 dB, 0.04 (8.0%), and 0.20 (12.0%), respectively. The standard deviations of the PC_PBF method are smaller than those of the PC algorithm in all situations. These results show that multichannel fusion can enhance the reconstruction performance and the computational stability. The typical evaluation results for killer whale and sonar are also shown in Table 2. The enhancement in SegSNR and NSEC is obvious.

Second, artificial audio signals from sonar, torpedo, and other natural audio signals from a dolphin, sea turtle, and killer whale are used as signal sources and experimented upon. The results show that all these signals can be correctly reconstructed.

To further testify to the performance, audio signals of the humpback whale and killer whale are mixed together to drive the underwater speaker. The NSEC and segmental SNR value of the reconstructed mixed signal is $(0.50\pm 0.02)$ and $(-3.44\pm 0.46)\mathrm{dB}$, respectively. The NSEC and SegSNR value of the hydrophone recorded signal is $(0.65\pm 0.03)$ and $(-2.38\pm 0.41)\mathrm{dB}$, respectively. The waveforms of the recovered signal and the hydrophone recorded signal are similar to the original mixed signal. From the evaluation results of NSEC and SegSNR, the signal recorded by the hydrophone is better than that reconstructed from speckle. The NSEC scores of the reconstructed mixed signal are $0.37\pm 0.02$ and $0.34\pm 0.02$ for the killer whale and humpback whale, respectively. The audio signal can be clearly distinguished as a mixed signal of humpback whale and killer whale signals for human ears.

The processing platform is a personal computer with a 3.20 GHz CPU (AMD Ryzen 5800H),16 GB of RAM, and a 4 GB graphics card (NVIDIA GeForce GTX 1650). The identifying process of 20 signals lasts 2.05 s, which shows that this method can identify underwater objects in real time.

In the experiment, aluminum foil is placed on the water surface to increase the scattering flux. This foil restricts the application of the sensing system to the laboratory environment. As long as the angle between the laser incident direction and the normal of the aluminum foil is smaller than 20 deg, the system performance is acceptable. Without the aluminum foil, the refraction and reflection of light should be considered.

The minimum detectable SPL at the water surface decides the application of the presented system. For the commercial Polytec OFV 353 sensor, the minimum detectable SPL is 120 dB re µPa^[1]. For the sensing system presented in this paper, experimental results show that the minimum detectable SPLs are 131.5 and 132.1 dB re µPa for 250 Hz and 2.5 kHz sinusoid signals, respectively. In a real marine environment, the SPL of sea turtles usually lies around 120 dB re µPa^[40], which is beyond the sensing range of the system presented in this paper. For cross air–water communication, the SPL of transducers is around 167 dB re µPa^[1], which lies in the detectable range of the sensing system presented in this paper. With the increase of the distance between the sensor and the vibrating object, the vibrating amplitude can be magnified through a defocused image. Therefore, the sensitivity can be increased with the increase of measuring distance.

The following parameters, such as water turbidity, water fluctuations, and ambient light, all affect the measurement performance of the sensor. For the present setup with aluminum foil, the effect of water turbidity can be ignored. A multibeamlet scheme can be used to suppress the data drop-out problem caused by water fluctuation. The interference caused by environmental factors (such as wind and passing ships) is low-frequency vibration (below 3 Hz^[13]) and can be filtered through a high-pass filter in the frequency domain. The effect of ambient light can be suppressed through an optical bandpass filter, which only transmits signal light to the image sensor.

The sensing performances are compared in Table 3. From Table 3, we can see that the proposed method has a wider viewing angle, lower detectable SPL, and longer sensing distance. Furthermore, the system setup proposed in this paper consists only of a laser source and a camera, which makes it compact and robust. If employing the multibeamlet scheme, simultaneous multipoint detection can be realized. As a comparison, the LDV scheme is principally a single-point method. Compared with the hydrophone method, the advantages of the proposed method are easy to deploy and suitable for large-range surveillance. The weakness of the proposed method is that its present performance is inferior to that of a hydrophone, the technique’s maturity has not reached the requirement of practical application, and the effect of environmental noise must be considered and suppressed.

In conclusion, a cross air–water interface underwater acoustic sensing system based on a linear array camera imaging is constructed. Phase correlation and a PBF multichannel fusion algorithm are introduced to extract and enhance the vibration signals. 1) Underwater acoustic signals with a frequency range from 200 Hz to 20 kHz are successfully detected with a relative frequency error of less than 0.5%. The minimum detectable SPL is 131.5 dB re µPa at 250 Hz. 2) Natural audio signals, including dolphins, turtles, killer whales, humpback whales, and artificial audio signals, including sonar and torpedoes, are tested and correctly reconstructed. For the reconstructed humpback whale audio signal, the NSEC and segmental SNR are $(0.52\pm 0.02)$ and $(-2.28\pm 0.13)\mathrm{dB}$, respectively. 3) Mixed humpback whale signals and killer whale signals are correctly reconstructed with NSEC better than $0.50\pm 0.02$.