Three-dimensional catadioptric vision sensor using omnidirectional dot matrix projection

Fuqiang Zhou; Xin Chen; Haishu Tan; Xinghua Chai

doi:10.3788/COL201614.111403

Abstract

In order to solve the problem of low measurement accuracy caused by uneven imaging resolutions, we develop a three-dimensional catadioptric vision sensor using 20 to 100 lasers arranged in a circular array called omnidirectional dot maxtric projection (ODMP). Based on the imaging characteristic of the sensor, the ODMP can image the area with a high image resolution. The proposed sensor with ODMP can minimize the loss of the detail information by adjusting the projection density. In evaluating the performance of the sensor, real experiments show the designed sensor has high efficiency and high precision for the measurement of the inner surfaces of pipelines.

In computer vision systems, omnidirectional three-dimensional (3D) measurements are an attractive area that can observe omnidirectional scenes simultaneously^[1,2]. The imaging model of catadioptric vision systems have been studied in-depth in recent years. Geyer and Daniilidis^[3] derived the geometric model of the catadioptric camera system, which has been used by many research works in the area of visual serving. Scaramuzza et al.^[4,5] gave a detail analysis of the catadioptric camera’s imaging model with a conic reflector. Similarly, vision sensors based on the catadioptric vision system also have achieved remarkable results. Paniagua et al.^[6] designed a wearable catadioptric vision sensor with a low-cost conic pattern laser in hand. Shin^[7] proposed an omnidirectional ranging system using a line structured light image to obtain all directional distance information effectively. Harmat^[8] presented an omnidirectional structure light sensor that could be applied to small, unmanned aerial vehicles, which can operate in a number of different environments. Zhang^[9] used light rings with different angular frequencies and intensities to build a catadioptric vision system for robotic navigation. Catadioptric camera systems based on structured light 3D vision measurements have the widest applications^[10–13] in practical inspections due to their fast measuring speed, non-contact, low cost, and robust nature in real measurements^[14–16]. However, although the working method and principle of the existing catadioptric structured light vision sensors are similar, the lasers used in the sensors are different, such as circular structured light and line structured light^[17], which may cause the loss of detail information in a local, highly reflective pipeline. The measurement accuracy is influenced by the uneven imaging resolution caused by the structure characteristics of the reflector and the projection density of the lasers.

To solve these problems, we present a 3D catadioptric vision sensor that can be applied to the measurement of a long pipeline. The omnidirectional dot maxtric projection (ODMP), which consists of 20 to 100 dot lasers and is positioned in 2 to 5 layers between the mirror and camera by 10° to 50°, can image a high imaging resolution area. The density of the ODMP can be adjusted using the measurement accuracy. In the following section, we describe how we can realize the measurement model by using the ODMP.

To measure the detail information of the inner surface, we first established the structure of the sensor. Figure 1 shows the optical path designation of the ODMP sensor. The camera placed below the conic reflector and the camera optical axis coincide with the symmetry axis of the conic reflector in the vertical direction. As is shown in Fig. 1, there are some identifiers that need introducing: the entrance pupil of the lens is located at $ξ$ unite above the origin. $h$ is the distance between the optical center of the camera and the vertex of the conic reflector. The working height of the ODMP is $m$ . The working distance of the novel sensor is set as the minimum distance from the light spots in the omnidirectional image to the camera optical axis, which is defined as $l$ .

Sign up for Chinese Optics Letters TOC. Get the latest issue of Chinese Optics Letters delivered right to you！Sign up now

Figure 1.Structure design of ODMP sensor.

In order to make the system compact and choose the appropriate lens, the field of view (FOV) angle $θ$ is determined by the focal length of the lens and the CCD camera size, which is $d$ : $θ = \tan^{- 1} (d / 2 f) .$ (1)Then, we define the point on the reflector surface^[4] as $S_{r} (t) = [z (t), γ (t)]$ : ${\begin{matrix} z (t) = t \\ γ (t) = \sqrt{(e^{2} - 1) \cdot t^{2} + 2 p t - p^{2}} \end{matrix},$ (2)where $e$ is the eccentricity and $p$ is the focus of the conic section. From Fig. 1, given one point $P$ in 3D space, the coordinates of $P$ are $(m - ξ, l)$ . The vectors along the incoming ray and reflected ray are given by ${\begin{matrix} V_{i} (t) = [(m - ξ) - t, l - \sqrt{(e^{2} - 1) \cdot t^{2} + 2 p t - p^{2}}] \\ V_{r} (t) = [t + ξ, \sqrt{(e^{2} - 1) \cdot t^{2} + 2 p t - p^{2}}] \end{matrix} .$ (3)

Because of the singularity of the point on the caustic, the determinant of the Jacobian $J (S_{r} (t) + r_{c} \cdot V_{i} (t))$ must vanish. For the class of conic catadioptric cameras, we can derive $r_{c}$ using $r_{c} = \frac{\dot{S_{r}} {(t)}_{γ} V_{i} {(t)}_{z} - \dot{S_{r}} {(t)}_{z} V_{i} {(t)}_{γ}}{\dot{V_{i}} {(t)}_{z} V_{i} {(t)}_{γ} - \dot{V_{i}} {(t)}_{γ} V_{i} {(t)}_{z}},$ (4)where $r_{c}$ is the distance of the scene point from the reflector. Since we know the geometry of the reflector, its surface normal $N_{r} (t)$ can be derived analytically. Reflecting $V_{r} (t)$ and incoming $V_{i} (t)$ are symmetrical about the surface normal $N_{r} (t)$ . We can derive the working height of laser projector as $m = ξ + t - \frac{γ (t) (γ (t) - l) (ξ + t (3 - 2 e^{2}) - 2 p)}{e^{2} t (t (e^{2} - 1) + 2 p)} .$ (5)

As is shown in Fig. 2, the height of the reflector is the depth of field $Δ L$ . $Δ L_{1}$ and $Δ L_{2}$ are the far depth of field and the near depth of field, respectively. $δ$ is the diameter of the circle of confusion. $F$ is the lens aperture. The value of $P^{'}$ in the $z$ -axis set is the focus of the curved mirror. Because the reflector needs to occupy the overall FOV of the camera, we can design the shape of the reflector, and the formula to compute $Δ L$ is $Δ L = Δ L_{1} + Δ L_{2} = 2 f^{2} F δ L^{2} / f^{4} - F^{2} δ^{2} L^{2} .$ (6)

Figure 2.Principle model of reflector designation.

Then, we analyze the effect of such a view surface on the resolution characteristics of the sensor. As is shown in Fig. 3, an infinitesimal area $δ A$ with the angle $ψ$ in the image plane projects into the reflector as a region of area $δ S$ through the entry pupil of the lens. Let an infinitesimal solid angle between this area image and the scene be $δ ω$ . The resolution of this sensor is defined as $δ A / δ ω$ . Thus, the solid angle visible at the infinitesimal region $δ A$ and the resolution of the image system is an inversely proportional relationship^[5].

Figure 3.Resolution model of multi-dot-field local reconstruction sensor.

The distance from the image point to the entrance pupil corresponding to $δ A$ is $f / \cos (ψ)$ . The solid angle subtended by the foreshortened area $δ A \cos (ψ)$ at the entrance pupil of the lens is $δ ϑ = δ A \cos^{3} (ψ) / f^{2} .$ (7)

Then, the area $δ S$ projected into the reflector by $δ A$ is derived using $δ S = δ A \cos (ψ) {(ξ + z (t))}^{2} / f^{2} \cos (ϕ),$ (8)where $ϕ$ is the angle between the reflected ray for the area $δ A$ and the surface normal at area $δ S$ . Then, the incoming angle of the scene point is $δ ω = δ S / r_{c}^{2}$ . The resolution for the omnidirectional catadioptric vision system can be obtained using $δ A / δ ω = f^{2} r_{c}^{2} (t) / [\cos (ψ) {(ξ + z (t))}^{2}] .$ (9)

We now present the model of measurement accuracy for the ODMP sensor. Using Fig. 4, some known geometry coordinates are defined in the catadioptric model, such as $O_{w} - X_{w} Y_{w} Z_{w}$ , $o_{c} - x_{c} y_{c} z_{c}$ , and $o_{μ} - x_{μ} y_{μ}$ . Given world point $P$ , $P^{'}$ is the point projected on the reflector, and the point in the normalized camera coordinate is $p^{'}$ . Their relationships are derived based on the model of the reflector’s surface: $p^{'} = [x y z]^{T} = [\begin{matrix} μ \end{matrix} ν \begin{matrix} S_{r} (ρ) \end{matrix}]^{T}, ρ = \sqrt{μ^{2} + ν^{2}},$ (10) $P^{'} = [X Y Z]^{T} = λ p^{'} = λ [μ ν \begin{matrix} S_{r} (ρ) \end{matrix}]^{T},$ (11)where $λ$ is the constant that describes the relationship between points $p^{'}$ and $P^{'}$ . Since the points $P$ and $P^{'}$ are collinear along the incoming ray $V_{r}$ direction, the relationship of $p^{'}$ , $P^{'}$ and $P$ can be obtained using ${\begin{matrix} P^{'} = p R + T \\ P^{' T} V_{r} = P^{T} V_{r} \end{matrix},$ (12)where $R$ and $T$ are the $3 \times 3$ rotation matrix and the $3 \times 1$ translation vector, respectively.

Figure 4.Measurement model of multi-dot-field local reconstruction sensor.

Then, we established a dot matrix plane $π$ to test the accuracy of the dot matrix projection using the ODMP. The point $P_{i}$ satisfies the equation of the dot matrix plane $π$ , $π P_{i} = 0 (i = 1, 2, \dots, n)$ . Then, the equation of the dot matrix plane $π$ can be determined using $A x + B y + C z + D = 0,$ (13)where $A - D$ are the coefficients of the dot matrix plane equation. Therefore, the 3D information of the dot matrix plane can reconstructed by computing the coordinates of $P_{i}$ : $π^{T} P = 0, P^{T} V_{r} = 0 .$ (14)

In summary, the overall measurement system using the ODMP and described through Eqs. (1)–(14) is as follows: (1)Check the structure of the catadioptric structured light vision sensor using Eqs. (1)–(6);(2)According to the structure of the sensor, finding the best image resolution of the non-central catadioptric system from Eqs. (7)–(9);(3)Verify the measurement accuracy of the sensor using the ODMP values from Eqs. (10)–(14).

We now apply results to find the best mirrors fitted with the 3D catadioptric vision sensor and study their resolution characteristics to see which can get the highest resolution.

In order to verify the accuracy of the 3D catadioptric vision sensor, let the working distance in the horizontal direction of the sensor be $l = 75 mm$ , and the working distance in the vertical direction of the sensor is $L = 80 mm$ . We select a high-speed camera whose focal length is $f = 6 mm$ . Based on Eqs. (1)–(7), the depth of field $Δ L$ is $Δ L = 19.85 mm$ . The detail configuration based on the designed model is shown in Table 1. In order to find the right mirror shape, we analyzed three commonly used mirrors: a hyperbolic mirror, a parabola mirror, and an ellipse mirror.

Category	Parameters
Camera	Interface: GigE vision	Resolution: 2456(H)×2058(V)
	Pixel size: 3.5 (μm)×3.5 (μm)
	Optic size: (2/3) in.	Frame rate: 15 fps
Lens	Focal length: 12 mm	Lens aperture F: 10 mm
	Horizontal angle of view: 49.25°
	Dimensions: 29.5 mm(D)×28.5 mm(L)
	The diameter of circle of confusion: 0.033 mm
Center dot projector	Size: ϕ12×35 mm	Wavelength: 650 nm
	Operating voltage: DC 5 V
	Fan angle: 90°	Output power: 2.5 mW

Table 1. Configuration of Multi-Dot-Field Local Reconstruction Sensor

View all Tables

In Fig. 5, we chose three results in different shapes with the best effects ( $ξ = 0$ ). For the ellipse reflector, the resolution drops drastically. However, the resolution characteristics of the hyperbola and parabola reflectors gradually change towards the edge of the reflector, and the resolution of the hyperbola is better than that of the parabola.

Figure 5.Resolution characteristics of the three reflectors.

Considering the class of reflector models analyzed in Fig. 5, we choose the hyperbola reflector. Given the hyperbola reflector, we can fit the right parameters in the model $(e, p, h)$ based on extreme points on the caustic and choose a projection area that has a high resolution that is better than the parabola. The designation results of the hyperbola mirror are shown in Table 2, and Fig. 6 shows the shape of the designed hyperbola mirror.

Figure 6.Shape of designed hyperbola reflector ( $ξ = 0$ ).

Hyperbola reflector	Eccentricity e: 1.03
	the focus of the conic section p: 120 mm
	the reflector profile: z2/1870.2165−x2+y2/1729.8112=1
	Distance to camera optical center h: 60 mm

Table 2. Configuration of Hyperbola Mirror

View all Tables

Then, based on the above analysis, the final projection area, calculated by the change of the incident angle, needs to be in the best position for projection. The resolution drops drastically when the incident angle is larger than 120°, as shown in Fig. 7.

Figure 7.Influence of angle of view field with resolution.

Therefore, combining the results in Figs. 5 and 7, the best area for projection can be obtained and the imaging area with the highest resolution is [7.25, 22.32]. The subsurface radius of the hyperbola reflector is at least 25 mm to guarantee image quality. Then, we considered the camera and reflector as a whole and selected three wavelengths ( $ω_{1} = 0.486$ , $ω_{2} = 0.587$ , and $ω_{3} = 0.656$ ) to verify the image resolution quality. Then we applied the Rayleigh criterion and Strehl’s theory to analyze the image quality of the ODMP.

Figure 8 shows the deformation degree of an emergent wave surface. In Fig. 9, the abscissa is the enclosed circle radius of the Gaussian image point, and the ordinate is the energy included in the enclosed circle. The fraction of enclosed energy is larger than 0.8, so the process of imaging is complete. The modulus of the optical transfer function are applied to test the resolution quality of the whole imaging process, as is shown in Fig. 10.

Figure 8.Wavefront aberration of the reflector.

Figure 9.Fraction of enclosed energy.

In Fig. 10, we can see that both the high frequency and low frequency parts have a high contrast. In the imaging system, we can obtain images with high resolutions.

Last, the accuracy of the sensor is verified by comparing two values, as described previously. One is the basic value defined as the position of the calibration target from the scene, and the other is a measurement value defined as the fitting surface, which is the dot matrix plane using the dot matrix projection, as shown in Fig. 11.

Figure 10.MTF result of designed reflector.

Figure 11.Measurement model of accuracy.

Then, we placed the target at a random position for local imaging and calculated the error of the designed sensor. Those data can apply to the verification of the sensor measurement accuracy because they are not used for the calibration.

Figure 12 shows the experimental image with the ODMP; we placed the calibration target at four different positions. The fitting calibration target can be reconstructed using the $5 \times 5$ dot matrix projector. The experimental results are shown in Fig. 13.

Figure 12.Experimental image.

Figure 13.Fitting calibration target in four place using $5 \times 5$ ODMP.

Considering the fitting calibration target reconstructed by the dot matrix projection, we analyzed the RMS error with the real calibration target, which is calculated by the process of traditional catadioptric calibration^[4], as shown in Fig. 14. The RMS error of the fitting calibration target is decreased with the increase of the dot matrix.

Figure 14.RMS error with the change of ODMP.

Then, we collected an image ( $5 \times 5$ ODMP) used an ordinary hyperbola reflector to compare the proposed sensors. The compared errors in the $x$ , $y$ , and $z$ directions between the proposed sensor and the ordinary are shown in Figs. 15 and 16.

Figure 15.Compared measurement error in $x$ , $y$ , $z$ directions.

Figure 16.Compared results in $x$ , $y$ , $z$ directions ( $5 \times 5$ ODMP).

In Fig. 15, the fluctuations of the errors in the $x$ , $y$ , and $z$ directions are small and have good robustness. Moreover, the proposed measurement model has high accuracy in the $x$ , $y$ , and $z$ directions, as shown in Fig. 16.

In conclusion, the presented sensor consists of a traditional perspective camera, a hyperbola reflector, and the ODMP. Its mathematical geometric senor model is constructed. The model is simple. Moreover, by analyzing the characteristics of the conic reflector, we obtain an area with a high resolution to decrease the difference of resolution in the reflector surface. The key of the catadioptric vision sensor is that it can minimize the loss of detail information in the process of measurement. The density of the ODMP can adjust to the different requirements in the measurement procedure. The encouraging results prove that this sensor is feasible for accurate catadioptric structure light vision measurement applications.

References

[1] F. Zhou, B. Peng, Y. Cui, Y. Wang, H. Tan. Opt. Laser Technol., 45, 1(2013).

[2] Z. Xiang, X. Dai, Y. Zhou, X. Gong. Meas. Sci. Technol., 25, 085005(2014).

[3] C. Geyer, K. Daniilidis. Int. J. Comput. Vis., 2, 766(2003).

[4] R. Swaminathan, M. Grossberg, S. Nayar. Int. J. Comput. Vis., 66, 211(2001).

[5] D. Scaramuzza, A. Martinelli, R. Siegwart. Fourth IEEE International Conference on Computer Vision Systems, 45(2006).

[6] C. Paniagua, L. Puig, J. J. Guerrero. Sensors, 13, 13903(2013).

[7] J. Shin, S. Y. Yi, Y. J. Hong, J. H. Suh. Journal of Institute of Control, Robotics and Systems, 16, 751(2010).

[8] A. Harmat, I. Sharf. Canadian Conference on Computer and Robot Vision (CRV)(2014).

[9] C. Zhang, J. Xu, N. Xi, Y. Jia, W. Li. International Conference on Advanced Intelligent Mechatronics IEEE/ASME, 43(2012).

[10] B. Bacca, X. Cufi, J. Salvi. IET Computer Vision, 7, 135(2013).

[11] F. Zhou, X. Chai, T. Ye, X. Chen. Chin. Opt. Lett., 13, 121501(2015).

[12] B. Wu, T. Xue, T. Zhang, S. Ye. Meas. Sci. Technol., 21, 283(2010).

[13] B. Wu, F. Zhang, T. Xue. Measurement, 61, 263(2015).

[14] F. Zhou, Y. Cui, B. Peng, Y. Wang. Opt. Laser Technol., 44, 1840(2012).

[15] B. Dong, R. Wang. Chin. Opt. Lett., 14, 031406(2016).

[16] O. Xu, S. Lu, S. Feng, S. Jian. Chin. Opt. Lett., 6, 818(2008).

[17] B. Wu, Y. Zhang. Adv. Mech. Eng., 5, 587904(2013).