Shape Measurement of a Sewer Pipe Using a Mobile Robot with Computer Vision

Aug 29, 2013

Inspection

By Kikuhito Kawasue and Takayuki Komatsu

Introduction

Sewer pipes totalling over 400,000 kilometres in length are buried under the ground in Japan. The number of worn pipes has increased and a number of subsidence accidents due to collapse have occurred. Since 80% of these pipes have a diameter of 450 mm or less, human inspectors cannot enter these pipes. Recently, pipe inspection has been performed using a moving cart with a CCD camera [1‐3].

In these systems, human inspectors judge the condition of the pipe based on images recorded by the CCD camera. Note that human judgment results in individual variations because the judgment depends upon individuals’ senses ‐ quantitative measurement is required. Automatic detection methods of cracks in a pipe have been introduced as prior work [4‐5], but the automatic shape measurement of the pipe is also required to evaluate the pipe’s condition.

In order to address this problem, we have developed a system equipped with two lasers, directed in different directions. The circular laser streaks that appeared on the inner surface of the pipe show the cross‐sectional shape of the pipe. The concept behind the proposed system is based on the method of the shape from the structure light [7‐10]. In this method, projecting a laser slit onto a three‐dimensionally‐shaped surface produces a line of illumination that appears distorted from other perspectives to that of the laser projector and can be used for an exact geometric reconstruction of the surface shape.

The cross‐sectional shape is detected by the proposed system using this method. The basic idea of the developed system is described in the patent disclosure [6]. Two parallel lasers are used in the system published in the patent disclosure. In our new approach, two laser slits are projected in different directions and the accuracy has been improved.

The robot moves along the axis of the tubular structure and the measured cross‐sectional shape data is accumulated to reconstruct the 3D structure on the computer. Two laser slits are projected from the proposed system in different directions and the shapes of the streaks that appear on the surface of the object are deformed depending on the tilt of the system. The tilt of the system with respect to the sewer pipe can also be measured by detecting the deformation of the streaks.

Therefore, the measurement of the vertical cross section of the pipe is possible, regardless of the tilt of the system, by considering the tilt of the system with respect to the target. Quantitative and quick measurement can be realized using the proposed system.

Measurement system

A photograph of the robot used for sewer pipe measurement is shown in Figure 1. The robot is controlled by a computer and driven by a pulse motor. A laser projector is placed on the front of the robot and the laser is projected in the radial direction using a corn‐shaped mirror. Laser ring streaks are displayed on the inner surface of the pipe. The streaks are recorded by the CCD camera and the cross‐sectional shape of the inner surface of the pipe is detected. Figure 2 shows the laser streaks that appear on the inner surface of the sewer. The streaks are deformed depending on the condition of the sewer pipe.

Fig. 1: Mobile robot for sewer pipe inspection [Source: University of Miyazaki]

Fig. 2: Laser streaks on the inner surface of the pipe [Source: University of Miyazaki]

The robot moves along the axis of the pipe and the detected cross‐sectional data is accumulated considering the amount of movement of the robot. An image of the reconstructed pipe displayed on a computer is shown in Figure 3.

Figure 4 shows the laser projector setup of the proposed system. Two laser projectors ‐ referred to hereinafter as Laser A and Laser B ‐ are installed on the robot. Laser A is projected perpendicular to the direction of movement of the robot and Laser B is tilted with respect to the direction of movement of the robot. The image recorded by the CCD is digitized through an IEEE1394 interface and processed by a computer.

Fig. 3: Reconstructed pipe through the accumulation of measured sections [Source: University of Miyazaki]

Fig. 4: Laser projector setup [Source: University of Miyazaki]

Figure 5 shows a photograph of the laser projector with a corn‐shaped mirror. A laser is reflected on the surface of the corn‐shaped mirror and is projected circumferentially. The laser draws a ring‐shaped streak on the inner surface of the pipe. The shape of the streak varies depending on the surface condition and the deformation of the pipe. The robot runs along the inside of the sewer pipe and measures the shape of the pipe by analysing the laser streaks that appear on the inner surface of the pipe.

Fig. 5: Laser projector with a corn‐shaped mirror [Source: University of Miyazaki]

Figure 6 shows the measurement section of the pipe onto which two lasers have been projected. When the system is tilted with respect to the pipe, the shape of the streak drawn by Laser B is deformed compared with the measurement section of Laser A. This means that the tilt of the system can be detected sensitively from the streak of Laser B. This enables accurate measurement considering the tilt of the system with respect to the pipe.

Fig. 6: Measurement section of the pipe [Source: University of Miyazaki]

In order to determine the tilt angle of Laser B, the measurement accuracy was checked while varying the tilt angle of Laser B in 5° increments. The results are shown in Figure 7. At approximately 35°, the error was less than 0.5°. Therefore, Laser B was set to have a tilt of 35° in the proposed system.

Fig. 7: Measurement section of the pipe (diagram) [Source: University of Miyazaki]

Fig. 8: Image captured by the robot [Source: University of Miyazaki]

Figure 8 shows the original image of the streaks captured by the CCD camera on the robot. Two streaks are captured simultaneously in a single image. Each streak can be separated easily by the following method. The streaks are recorded as the shape of an ellipse, the centre of which is quickly estimated as follows.

  1. The streak is divided into short segments.
  2. The angles of all the segments are calculated.
  3. A pair of segments for which the orientations are anti‐parallel (180°) is found. The midpoint of the line joining the segments will be situated at the centre of the ellipse, as shown in Figure 9.
  4. The midpoints on the parameter plane are accumulated for all pairs of corresponding segments.
  5. The average centre‐point of the ellipses is determined by finding the peak point on the parameter plane.
  6. The distance of each streak pixel from the centre is calculated as shown in Figure 10, and each streak is separated by determining the separation line on the graph. This enables the robust detection of the laser streaks.

Fig. 9: Centre detection of the laser streaks [Source: University of Miyazaki]

Fig. 10: Separation of the streak [Source: University of Miyazaki]

Calibration of the system

Calibration [11‐13] constitutes an important task in realizing the quantitative measurement. In order to calculate the quantitative data from an image, the following calibration procedure is performed. Since the size of the pipe usually has a range from 200 mm to 600 mm, a wide lens is required in order to satisfy the range requirement for the pipe. The distortion that generally accompanies wide lenses should be corrected through pre‐processing. The relation between the original camera coordinates and the undistorted coordinates is defined [14‐19] as follows:

cases u ' = k 1 ( 1 + k 2 ( u 2 + v 2 ) ) u v ' = k 1 ( 1 + k 2 ( u 2 + v 2 ) ) v

Formula 1

where (u,v) are the original camera coordinates and (u’,v’) are the undistorted camera coordinates. The origin of these coordinates is at the centre of the image. The image can be corrected as shown in Figure 11 by finding adequate parameters (k1,k2).

A standard pipe with a known size is used for the calibration. The laser streak recorded by the CCD camera is used for calibration. More than four pairs of camera coordinates and global coordinates of the laser position are inputted into the computer to establish the calibration.

Fig. 11: Correction on the camera distortion [Source: University of Miyazaki]

The relation between the camera coordinates (u,v) and the global coordinates (x,y) can be defined as follows:

s [ u v 1 = b m a t r i x k 11 k 12 k 13 k 21 k 22 k 23 k 31 k 32 1 [ x y 1 ] ]

Formula 2

where k11 through k32 are parameters that consider the rotation, scale, and displacement between the camera coordinates and the global coordinates. These parameters are determined by inputting corresponding pairs of camera coordinates and global coordinates. The inputs are performed using a mouse to designate the camera coordinates and a keyboard to designate the corresponding global coordinates. Here, k11 through k32 are determined by inputting more than four pairs of corresponding coordinates into Eq. [2]. The function for converting from the camera coordinates to the global coordinates can be written as:

[ x y = b m a t r i x k 31 u − k 11 k 32 − k 12 k 31 v − k 21 k 32 − k 22 − 1 [ k 13 − u k 23 − v ] ]

Formula 3

All of the points on the circular laser streak are converted into global coordinates and the cross‐sectional shape of the pipe is estimated. The overall shape of the pipe is reconstructed by integrating the cross‐section of the pipe along the axis of the pipe while considering the movement of the robot. Since the robot is driven by a pulse‐motor, the distance driven can be estimated based on the number of pulses sent to the pulse‐motor.

Detection of the tilt of the robot against the axis of the sewer pipe

Figure 12a shows the streak detected by scanner B when the system rotates around the x‐axis. In this figure, l is the length from the bottom of the measured structure to the peak (Point P). The tilt of the system with respect to the x‐axis is detected by measuring the change of l, which is sensitive to the tilt around the x‐axis. The curve of Figure 12b shows the streak detected by scanner B when the system rotates around the y‐axis. When the system rotates around the y‐axis, the streak detected by laser B also rotates according to the system tilt itself. Here, θb is the angle between OP and the axis perpendicular to the bottom of the structure. The tilt of the system with respect to the y‐axis can be detected by the change in θb. The tilt with respect to the z‐axis can be detected by the tilt of the bottom line of the detected section.

Fig. 12: Correction on the camera distortion (scheme) [Source: University of Miyazaki]

When the robot is tilted with respect to the pipe, as shown in Figure 13, the robot measures the area of [I] but the robot misinterprets the area to be [II] without considering the tilt with respect to the pipe. Therefore, the system corrects the data obtained by considering the tilt. The formula for the correction can be written as follows:

[ Z ' Y ' = b m a t r i x cos θ − sin θ sin θ cos θ [ Z Y ] ]

Formula 4

Fig. 13: Measurement area of the robot tilt [Source: University of Miyazaki]

In order to evaluate the performance of the system, a robot was placed on a turntable and was rotated during measurement, as shown in Figure 14. The laser is projected onto the surface of the pipe and the CCD camera detects the laser streaks that appear on the surface of the pipe. The measurement was performed from 0° to 10° at intervals of 2° and the measurement accuracy was investigated. The error was less than 0.6 mm for the rotation around each axis.

Fig. 14: Tilt detection experiment [Source: University of Miyazaki]

Detection of the tilt of the pipe with respect to the gravitational direction

In the previous section, the tilt correction of the robot with respect to the pipe is discussed. However, the tilt of the pipe with respect to the gravitational direction should also be considered. The subsidence of the ground causes the sewage pipes to bend, which causes sulphuric acid to accumulate in some cases, shortening the life of the sewage pipes, as shown in Figure 15.

Fig. 15: Bent pipe caused by a subsidence of the ground. Sulphuric acid accumulates and it shortens the life of the sewage pipe. [Source: University of Miyazaki]

Therefore, a function to detect the tilt of the pipe with respect to the gravitational direction is used to measure the bend of the pipe. An acceleration sensor (ART‐Promotions:WAA‐006) is used to detect the tilt with respect to the gravitational direction. The experimental setup is shown in Figure 16. The pipe was tilted by 3°,and the velocity of the robot movement was set at 10 mm/s. The shape of the pipe was measured by considering the tilt of the pipe. The reconstructed shape of the pipe is shown in Figure 17. The vertical displacement was calculated using the data obtained by the acceleration sensor. Figure 18 shows the vertical displacement with respect to the movement of the robot. The blue line shows the results of the experiment and the red line shows the ideal case. The error was approximately 0.1 mm in this experiment.

Fig. 16: Photograph of the tilt detection experiment with respect to the gravitational direction. [Source: University of Miyazaki]

Fig. 17: Experimental results of tilt detection of the pipe [Source: University of Miyazaki]

Fig. 18: Experimental results of tilt detection with respect to the gravitational direction. The blue line shows the results of the experiment and the red line shows the ideal case. [Source: University of Miyazaki]

Measurement Flow

The measurement flow is summarized as shown in Figure 19. The robot is driven by a pulse motor and runs along the pipe. The degree of movement of the robot can be detected by counting the number of pulses sent to the pulse motor. The CCD camera captures the laser streaks that appear on the inner surface of the sewer pipe. The detected streak in camera coordinates is converted to global coordinates. The tilt of the robot with respect to the pipe is calculated by the deformation of the streak, as mentioned earlier. The detected cross section of the pipe is accumulated in order to reconstruct the sewer pipe on the computer. The tilt of the pipe with respect to the gravitational direction is also detected using an acceleration sensor. This procedure is repeated until the robot stops.

Fig. 19: Measurement flow [Source: University of Miyazaki]

Experimental results of the shape measurement

When the measurement system is tilted with respect to the structure axis, the measurement section is also tilted with respect to the tubular structure. Therefore, the detected data must be rotated considering the tilt (rotation) of the measurement system with respect to the structure. Figure 20 gives an example of the point cloud data of the sewer pipe. The data was obtained while the system was intentionally swung forward to the right and the left. The left‐hand image does not take into consideration the tilt of the system. Although the pipe was originally straight, the pipe constructed on the computer is curved because the tilt of the system is not taken into consideration. In the right‐hand image of Figure 20, the data is corrected by considering the tilt of the system.

Fig. 20: Reconstructed sewer pipe displayed on the computer [Source: University of Miyazaki]

The tilt of the pipe with respect to the gravitational direction is also detected using an acceleration sensor, as stated in Section 5. The shape data with the tilt information is stored in the computer.

Another experimental result is shown in Figure 21. Clay letters ‘A’ and ‘B’ are attached to the pipe. The cross sections of the pipe are measured and summarized along the axis in order to reconstruct the shape of the sewer pipe on the computer. The reconstructed pipe and the letters using the measured data are shown in Figure 22.

Fig. 21: Inside of the measured sewer [Source: University of Miyazaki]

Fig. 22: Experimental pipe using the measured data [Source: University of Miyazaki]

The specifications of the sewer pipe are estimated by computer using the obtained point cloud data. The average error of the measurement is less than 0.4 mm.

Conclusion

A sewer pipe measurement robot that uses machine vision was introduced in the present paper. Quantitative measurement was realized using two lasers and an image processing technique. Two lasers enabled the detection of the tilt of the robot with respect to the axis of the pipe. The shape of the inner surface of the pipe is reconstructed by accumulating the measured sections along the axis of the sewer pipe. The average measurement error was approximately 0.5 mm, which is sufficient for the measurement of the sewer pipe. The condition of the sewer pipe can be evaluated quantitatively using the proposed system.

Acknowledgments

The present study was supported by a Grant‐in‐Aid for Scientific Research (18656153).

References

[1] Comittee, S.P.R.: Report of sewerage policy to realize circular path. Ministry of land, infrastructure and transport (2006)

[2] Sinha, S., Fieguth, P.: Morphological segmentation and classification of underground pipe images. (2006) Vol 17: 21‐31

[3] Kolesnik M., Barato, G.: Online distance recovery for a sewer inspection robot. (2000) Vol I: 504‐507

[4] Iyer, S., Sinha, S.: A robust approach for automatic detection and segmentation of cracks in underground pipeline images. (2005) Vol: 921‐933

[5] Fieguth, P., Sinha, S.: Automated analysis and detection of cracks in underground scanned pipes. (1999) Vol IV: 395‐399

[6] Kawasue, K. : Pipe measurement system. Japanese Patent Disclosure. (2006)

[7] Penney, C., Corby, N.: Coded aperture light detector for three dimensional camera. (1989)

[8] Wang, Y.: Characterizing three‐dimensional surface structure from visual images. (1991) Vol 13: 52‐60

[9] Kak, A., Boyer, K., Safranek, R., Yang, H.: Knowledge‐ based stereo and structured light for 3‐d robot vision. (1986) 185‐218

[10] Boyer, K., Kak, A.: Color‐encoded structured light for rapid active ranging. (1987) Vol 9: 14‐28

[11] Davies, E.R.: “Machine Vision: Theory, Algorithms, Practicalities”, Academic Press. 1990

[12] Torras, C. “Computer Vision: Theory and Industrial Applications”, SpringerVerlag. 1992

[13] Fargeras, O.: Three‐dimensional Computer Vision, A geometric viewpoint. 1996

[14] Tsuji, S., Matumoto, F.: “Detections of ellipses by a modified Hough transform”, IEEE Trans. Comput. 27, pp. 777‐781

[15] Tsai, R.A.: Versatile camera calibration technique for high‐accuracy 3d machinevision metrology using off‐ the‐shelf TV cameras and lenses. (1987) Vol 3: 323‐344

[16] Tsai, R.: Ancient and accurate camera calibration technique for 3‐d machine vision. (1986) 364‐374

[17] Gennery, D.: Generalized camera calibration including fish‐eye lenses. (2006) Vol 68: 239‐266

[18] Gennery, D.: Camera calibration including lens distortion. JPL internal report D‐8580 ,1991

[19] Goshtasby, A.: Correction of image deformation from lens distortion using Bezier patches. (1989) Vol 47: 385‐394

 

© 2013 Kawasue and Komatsu; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Contact

Department of Environmental Robotics, University of Miyazaki

Kikuhito Kawasue

Miyazaki

Japan

E-Mail:

kawasue@cc.miyazaki-u.ac.jp

© 2024 visaplan GmbH, a subsidiary of Prof. Dr. Ing. Stein & Partner GmbH

UNITRACC 3.10.15.1