JP6797607B2 - Image processing device, image processing method and program for image processing - Google Patents

Description

The present invention relates to a technique for specifying the position of a shadow reflected in an image.

For example, there is a technique for inspecting a road for cracks, deterioration, etc. by photographing a road as a moving image or a still image and analyzing the photographed image with software (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2011-179874

By the way, in the above technique, there is a problem that the shadow of the building and its edge (the boundary portion between the shadow and the non-shadow portion) due to the sunlight hinder the image analysis. First, since the shadow portion has low brightness, the resolution and accuracy of image analysis are lowered. If it is known that there is a shadow part, it is possible to perform an analysis that ensures accuracy by adjusting or analyzing an image suitable for the analysis of the shadow part. However, it is difficult to determine by software whether it is a shadow or not. Although it is possible to determine whether or not it is a shadow by visually recognizing it by a person, the amount of work to be performed by a person visually recognizing it increases the amount of work and reduces the meaning of using the image analysis technique.

In addition, there is a problem that the edge portion of the shadow is erroneously detected as a crack formed on the road surface. This problem can also be avoided by human eyes if it is confirmed by human eyes, but it reduces the meaning of using image analysis technology. Image analysis by software processing is to reduce the work done by humans as much as possible and to improve the work efficiency. Therefore, it is desirable to minimize the work that people see and judge.

Against such a background, an object of the present invention is to provide a technique for identifying the position of a shadow caused by sunlight in a captured image.

The invention according to claim 1 is an image data acquisition unit that acquires image data of an image of an object captured by a camera, a point cloud data acquisition unit that acquires point cloud data acquired by a laser scanner, and a position of the sun. A shadow area point cloud calculation unit that calculates a shadow area point cloud that projects the point cloud data from the sun onto an object, and a shadow area point cloud projection unit that projects the shadow area point cloud onto the image. It is provided with an overhead line detection unit that detects an overhead line based on the point cloud data, and the shadow area point cloud calculation unit calculates a point cloud of a portion that is a shadow of the detected overhead line and detects the overhead line. The image processing apparatus is characterized in that a process of predicting a position where the overhead lines are continuously present is performed based on a point cloud of the overhead lines .

The invention according to claim 2 is the coordinate conversion of the point cloud data acquired by the laser scanner into the ground coordinate system and the point cloud data described in the ground coordinate system in the invention according to claim 1. Is further provided with a coordinate conversion unit that converts the coordinates into the camera coordinate system, and the calculation of the shadow area point cloud is performed in the ground coordinate system. Here, the coordinate conversion of the point group data acquired by the laser scanner to the ground coordinate system is performed when the coordinate system handled by the laser scanner (scanner coordinate system) is directly converted to the ground coordinate system or from the scanner coordinate system. This includes both cases where the coordinates are converted to the ground coordinate system after the coordinate conversion to another coordinate system such as the IMU coordinate system is performed.

The invention according to claim 3 is the invention according to claim 1 or 2, wherein in the image, a shadow area extension that extends a shadow area around the shadow area point group based on a difference in the brightness value of a pixel depending on a position. It is characterized by having a part.

The invention according to claim 4 is characterized in that the invention according to claim 3 includes a contrast adjusting unit for adjusting the contrast of the shadow region projected on the image in the image.

The invention according to claim 5 is characterized in that, in the invention according to claim 3, a shadow boundary line specifying portion for specifying a shadow boundary line in the image is provided based on the expanded shadow region. ..

The invention according to claim 6 is the invention according to claim 5, based on the position of the cracked portion detecting portion for detecting the cracked portion to be imaged from the image and the boundary portion in the image. It is characterized by including a false detection determination unit for determining false detection of a cracked portion detected by the partial detection unit.

The invention according to claim 7 includes an image data acquisition step for acquiring image data of an image of an object captured by a camera, a point cloud data acquisition step for acquiring point cloud data acquired by a laser scanner, and a position of the sun. A shadow area point cloud calculation step for calculating a shadow area point cloud obtained by projecting the point cloud data from the sun onto the object, and a shadow area point cloud projection step for projecting the shadow area point cloud onto the image. It has an overhead line detection step for detecting an overhead line based on the point cloud data, and in the shadow area point cloud calculation step, a point cloud of a portion to be a shadow of the detected overhead line is calculated, and the overhead line is calculated. An image processing method characterized in that, in the line detection step, a process of predicting a position where the overhead lines are continuously present is performed based on a point cloud of the overhead lines .

The invention according to claim 8 is a program to be read and executed by a computer.
Based on the position of the sun, the image data acquisition unit that acquires the image data of the image of the object taken by the computer with the camera, the point cloud data acquisition unit that acquires the point cloud data acquired by the laser scanner, and the sun Based on the shadow area point cloud calculation unit that calculates the shadow area point cloud that projects the point cloud data on the object, the shadow area point cloud projection unit that projects the shadow area point cloud onto the image, and the point cloud data. It functions as an overhead line detection unit that detects an overhead line, the shadow area point cloud calculation unit calculates a point cloud of a portion that becomes a shadow of the detected overhead line, and the overhead line detection unit calculates the overhead line. This is an image processing program characterized in that a process of predicting a position where the overhead lines are continuously present is performed based on the point cloud of .

According to the present invention, it is possible to obtain a technique for identifying the position of a shadow caused by sunlight in a captured image.

It is a conceptual diagram of an embodiment. It is explanatory drawing of the principle of an embodiment. It is a block diagram of an embodiment. It is a flowchart which shows an example of the processing procedure. It is explanatory drawing explaining the shadow area point cloud. It is an image diagram of the camera image which projected the shadow area point cloud.

(Overview)
FIG. 1 shows the vehicle 100. The vehicle 100 includes a laser scanner 101, a camera 102, an IMU (Inertial Measurement Unit) 103, a GNSS device 104, and a storage device 105. The laser scanner 101 performs a laser scan of the surrounding environment and acquires point cloud data. As the laser scanner 101, the techniques described in JP-A-2010-151682, JP-A-2008-268004, and US Pat. No. 8,767190 can be used.

The camera 102 captures images at specific time intervals and obtains image data of the captured images. A type of camera 102 that captures a moving image can also be used. In this case, the frame image is used as the captured image. The IMU 103 is equipped with a 3-axis acceleration, 3-axis angular velocity, and 3-axis geomagnetic 9-axis motion sensor, and measures changes in attitude and inclination. The GNSS device 104 receives a navigation signal from a navigation satellite such as a GPS satellite, and measures the position based on the navigation signal. Further, the GNSS device 104 has a clock function that outputs highly accurate time information based on a navigation signal from a navigation satellite.

The storage device 105 is a data storage device composed of a hard disk device and a semiconductor memory. The storage device 105 mutually exchanges point cloud data acquired by the laser scanner 101, image data of an image taken by the camera 102, data on changes in posture and inclination measured by IMU 103, and position and time data measured by the GNSS device 104. Remember in the associated state. The external orientation elements (posture and position) of the laser scanner 101, camera 102, IMU 103 and GNSS device 104 with respect to the vehicle 100 are measured and known in advance.

(Explanation of principle)
The system of FIG. 1 is used to detect cracks in concrete paved road surfaces by image analysis. In this example, a technique for reducing the problem of false detection of cracks due to the influence of shadows not exposed to sunlight will be described.

The shadow in the present specification means a part formed by the sunlight hitting a building or the like and not directly hit by the sunlight. As shown in FIG. 2, when the shadow of the building is reflected on the road surface, a linear boundary line is generated. This boundary line is erroneously detected as a crack on the road surface. In this example, the three-dimensional position of the building is grasped by laser scanning, and the shadow area that is not exposed to sunlight is calculated from the three-dimensional information (point cloud data). Since the position of the sun on the celestial sphere at a specific time can be accurately obtained from astronomical data and calculation formulas, the position of the shadow reflected on the road surface can be obtained by geometric calculation.

Then, by projecting the position of the shadow on the road surface into the image captured by the camera 102, image data showing where the shadow is reflected in the image is created, and erroneous detection of cracks can be determined. In addition, since a shadow portion in the image can be specified in advance, the accuracy of detecting cracks in the shadow portion can be improved by adjusting the contrast of that portion.

(Structure of image processing unit)
FIG. 3 shows a block diagram of the image processing apparatus. FIG. 3 shows the image processing device 200. The image processing device 200 has a basic function as a computer. The image processing device 200 is a device that detects cracks on the road surface by image analysis. In particular, the shadow portion in the image is specified by using the principle shown in FIG. 2, and the accuracy of detecting cracks on the road surface is lowered due to the presence of shadows. Perform processing to suppress.

The image processing device 200 includes an image data acquisition unit 201, a point cloud data acquisition unit 202, a coordinate conversion unit 203, a shadow area point cloud calculation unit 204, a shadow area point cloud projection unit 205, a shadow area expansion unit 206, and a shadow area identification unit. 207, a contrast adjusting unit 208, a shadow boundary line specifying unit 209, a cracked portion detecting unit 210, an erroneous detection determination unit 211, an overhead line detecting unit 212, a storage unit 213, and a vehicle position specifying unit 214 are provided.

The functional unit included in the image processing device 200 described above may be configured by software or may be configured by a dedicated arithmetic circuit. In addition, a functional unit configured by software and a functional unit configured by a dedicated arithmetic circuit may coexist. For example, each functional unit shown in the figure is composed of electronic circuits such as a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), and a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array).

Whether each functional unit is configured with dedicated hardware or software-like configuration by executing a program on the CPU is determined in consideration of required calculation speed, cost, power consumption, and the like. For example, if a specific functional unit is configured by FPGA, it is advantageous in terms of processing speed, but the cost is high. On the other hand, a configuration in which a specific functional unit is realized by executing a program on a CPU can save hardware resources, which is advantageous in terms of cost. However, when the functional unit is realized by the CPU, the processing speed is inferior to that of the dedicated hardware. Further, when the functional unit is realized by the CPU, it may not be possible to handle complicated calculations. It should be noted that configuring the functional unit with dedicated hardware and configuring it as software are equivalent from the viewpoint of realizing a specific function, although there are the above-mentioned differences.

In this example, the image processing device 200 is composed of a computer such as a workstation that can process a large amount of data at high speed. Of course, if there is no problem in the processing capacity, the image processing apparatus 200 can be configured by using a commercially available personal computer. It is also possible to configure the system so that the functions of the image processing apparatus 200 are distributed and processed by a plurality of computers.

The image data acquisition unit 201 acquires image data of an image taken by the camera 102. The image data is data of a frame image constituting a still image or a moving image taken at a specific time interval. The point cloud data acquisition unit 202 acquires the point cloud data acquired by the laser scanner 101. The point cloud data is data that captures the scan target as a collection of points for which three-dimensional coordinates have been acquired.

The coordinate conversion unit 203 is related to the coordinate system that describes the point group data, and is involved in coordinate conversion from the scanner coordinate system to the IMU coordinate system, coordinate conversion from the IMU coordinate system to the ground coordinate system, and from the ground coordinate system to the IMU coordinate system. Coordinate conversion and coordinate conversion from the IMU coordinate system to the camera coordinate system are performed. The scanner coordinate system is a coordinate system originating from the laser scanner 101. The IMU coordinate system is a coordinate system with the IMU 103 as the origin. The horizontal coordinate system is a coordinate system fixed on the ground. The camera coordinate system is a coordinate system with the camera 102 as the origin.

The position of the origin and the direction of the coordinate axes of the scanner coordinate system, the IMU coordinate system, and the camera coordinate system change every moment as the vehicle 100 moves. Of course, if the vehicle 100 does not move, the origin of the coordinates does not move, and if the direction of the vehicle 100 does not change, the direction of the coordinate axes of the coordinate system does not change.

The coordinate conversion from the scanner coordinate system to the IMU coordinate system is performed based on the external orientation elements of the laser scanner 101 and the IMU 103 for the vehicle 100. More specifically, the coordinate conversion from the scanner coordinate system to the IMU coordinate system is performed as follows. First, consider a specific time t. The relative relationship between the scanner coordinate system and the IMU coordinate system at this time t can be seen from the external orientation elements of the laser scanner 101 and the IMU 103 with respect to the vehicle 100. That is, the scanner coordinate system can be converted into the IMU coordinate system by translating and rotating the scanner coordinate system at time t. The conversion coefficient at this time is obtained from the external orientation elements of the laser scanner 101 and the IMU 103 with respect to the vehicle 100. Therefore, the coordinate value of the specific point A in the scanner coordinate system at time t in the IMU coordinate system can be calculated. By performing the same process at an arbitrary time, the coordinate conversion from the scanner coordinate system to the IMU coordinate system at that time is performed.

The conversion from the IMU coordinate system to the horizontal coordinate system is performed using the information of the vehicle trajectory. Vehicle trajectory information is obtained from the IMU 103 and the GNSS device 104. More specifically, the coordinate transformation from the IMU coordinate system to the horizontal coordinate system is performed as follows. First, consider a specific time t. The relative relationship between the IMU coordinate system and the ground coordinate system at time t is obtained from the external orientation element of the IMU 103 with respect to the vehicle 100 and the position and orientation of the vehicle 100 at time t. Information on the position of the vehicle 100 is obtained from the GNSS device 104, and information on the orientation of the vehicle 100 is obtained from the IMU 103. In this case, the IMU coordinate system can be converted into the horizontal coordinate system by translating and rotating the IMU coordinate system at time t.

The conversion coefficient at this time is obtained from the external orientation element of the IMU 103 with respect to the vehicle 100 and the position and orientation of the vehicle 100 at time t. Therefore, the coordinate value of the specific point A in the IMU coordinate system at time t in the horizontal coordinate system can be calculated. Then, by performing the above processing at time t ₁ , t ₂ , t ₃ ..., Points acquired at each time at time t ₁ , t ₂ , t ₃ ... (Point cloud by a collection of these points) The data is composed) can be converted to the ground coordinate system.

The time interval in the coordinate conversion of the scanner coordinate system → IMU coordinate system → ground coordinate system is determined by the scan speed of the laser scanner 101. The scanning speed of the laser scanner 101 is, for example, about 1 kHz to 10 kHz. The narrower (shorter) the above time interval, the higher the accuracy, but the larger the amount of calculation.

The conversion of the point cloud data from the horizontal coordinate system to the IMU coordinate system is performed as follows. This conversion is performed at each time ticked at times T ₁ , T ₂ , T _3, .... The time Tn at this time is the shooting time of the camera 102. The shooting interval of the camera 102 is, for example, about 0.1 second to 0.5 second. Of course, shooting may be performed at shorter time intervals, but the load required for the calculation increases.

First, the position and orientation of the vehicle at the exterior orientation parameters and time _{T 1} of the IMU103 (this data is obtained from IMU103) obtaining a relationship between the ground coordinate system and the IMU coordinate system at time _{T 1} from. This allows the coordinate transformation to IMU coordinate system from the ground coordinate system of the point cloud data at time T _1, the point group data written in IMU coordinate system is obtained at time T _1. By performing this process at other times as well, point cloud data in the IMU coordinate system at the shooting time Tn can be obtained.

The coordinate conversion of the point cloud data from the IMU coordinate system to the camera coordinate system is performed based on the external control elements of the camera 102 and the IMU 102 by the same principle as in the case of the coordinate conversion from the scanner coordinate system to the IMU coordinate system. Since the IMU coordinate system and the camera coordinate system move together with the vehicle 100, the coordinate conversion from the IMU coordinate system to the camera coordinate system is performed by designating the time Tn, and is obtained at the designated time Tn.

Instead of converting from the scanner coordinate system to the IMU coordinate system to the ground coordinate system, it is possible to convert from the scanner coordinate system to the ground coordinate system without going through the IMU coordinate system. In either case, the result is a conversion from the scanner coordinate system to the horizontal coordinate system. Further, instead of converting from the ground coordinate system to the IMU coordinate system to the camera coordinate system, it is possible to convert from the ground coordinate system to the camera coordinate system without going through the IMU coordinate system. In this case, the result is a conversion from the horizontal coordinate system to the camera coordinate system.

Since the scanner coordinate system changes from moment to moment according to the movement of the vehicle in the ground coordinate system, it is possible to create a single set of data (point cloud) by converting each point to the ground coordinate system. The building point cloud (shadow area point cloud that constitutes the shadow of the building) can be projected from the direction of the sun to the road surface point cloud only after the point cloud of. The shadow region point cloud in the ground coordinate system thus obtained is converted into the IMU coordinate system at the time Tn at which the image was taken, converted into the camera coordinate system, and projected onto the image captured by the camera. In this method of once converting to the horizontal coordinate system, the point cloud data can be easily handled even if the scan timing of the laser scanner 101 and the shooting time of the camera 102 are not synchronized. Therefore, the degree of freedom in selecting the models of the laser scanner 101 and the camera 102 and setting the operation timing is high, and the system can be configured at low cost.

Hereinafter, an example of the processing performed by the coordinate conversion unit 203 will be described. First, consider a three-dimensional coordinate system (first coordinate system: scanner coordinate system) with the laser scanner 101 as the origin as the coordinate system of the point cloud data. On the other hand, consider a three-dimensional coordinate system (second coordinate system: camera coordinate system) with the camera 102 as the origin. Here, it is assumed that the external orientation element of the laser scanner 101 (position and orientation in the coordinate system handled by IMU 103) and the external orientation element of the camera 102 (position and orientation in the coordinate system handled by IMU 103) are known. Further, the three-dimensional coordinate system handled by the IMU 103 is defined as a third coordinate system (IMU coordinate system).

Since the external orientation element of the laser scanner 101 is known, the first coordinate system can be converted into the third coordinate system (IMU coordinate system) by rotating and translating the first coordinate system. Further, if the position and orientation of the IMU 103 at the acquisition time of each point acquired by the laser scanner are known in the fourth coordinate system (horizontal coordinate system), the point cloud data can be described on the fourth coordinate system. .. On the other hand, since the external orientation element of the camera 101 is known, the second coordinate system can be converted into the third coordinate system by rotating and translating the second coordinate system. Further, if the position and orientation of the IMU 103 at the shooting time are known in the fourth coordinate system, the position and orientation of the camera 102 at the time can be converted into the fourth coordinate system (ground coordinate system). As a result, it is possible to describe, for example, the direction of the optical axis of the camera 102 and the angle of view (shooting field of view) on the fourth coordinate system (horizontal coordinate system). Of course, it is possible to convert a point cloud in the horizontal coordinate system into a second coordinate system (camera coordinate system) by a mathematical operation.

The shadow area point cloud calculation unit 204 calculates a point cloud in a shadow area that is not exposed to sunlight as a shadow area point cloud due to the existence of the point cloud data based on the position of the sun. This process is performed on the horizontal coordinate system. By performing this processing on the horizontal coordinate system, the point cloud data can be handled as a set of data, the processing time is shortened, and the load on the processing hardware is reduced. As the sun moves, so does the shadow. Therefore, the calculation of the shadow area point cloud is repeated at specific time intervals. Specifically, the shadow region point cloud is calculated in synchronization with the shooting timing of the camera 102 (for example, every 0.1 seconds). In order to reduce the burden of calculation, it is also possible to calculate the shadow region point cloud at a rate of once every a plurality of shooting timings of the camera 102. For example, when the shooting interval is 0.1 second, it is possible to calculate the shadow region point cloud every 0.5 second.

Hereinafter, an example of calculating the shadow area point cloud will be described. For example, suppose that when the direction of the sun is viewed from a certain viewpoint A on the road surface, there is a building in that direction and the sun cannot be seen directly. In this case, there is a point B constituting the building on the line connecting the viewpoint A and the sun, and this point B is acquired as one of the point clouds when the laser scan is performed from the viewpoint. The point where this point B is projected onto the road surface with the sun as the viewpoint (origin) is one of the shadow region point clouds. This projection point coincides with the viewpoint A on the road surface. FIG. 5 shows an example of the relationship between the point cloud obtained by the laser scan and the shadow region point cloud.

In the process of calculating the shadow region point cloud, first, the point cloud acquired by the point cloud data acquisition unit 202 (the point cloud measured by the laser scanner 101) is projected onto the ground (road surface) using the sun at infinity as a light source. By this projection, a group of projection points on the ground corresponding to each point cloud measured by the laser scanner 101 is obtained as a shadow projection point cloud. The shadow projection point cloud is a point cloud composed of point-like shadows.

For the data on the position of the sun and its calculation program, for example, those obtained from the site of NASA's Jet Propulsion Laboratory are used. Of course, the position of the sun may be specified by using a unique measured value or a calculation program. For example, data on the position of the sun is described in the science chronology, and many calculation methods and programs based on it are open to the public. If the position and time of the viewpoint are known based on this information, the position of the sun on the celestial sphere can be accurately calculated.

Hereinafter, an example of the process of calculating the shadow area point cloud will be described. First, point cloud data in the horizontal coordinate system is generated by converting each point acquired by the laser scanner 101 from the scanner coordinate system (the first coordinate system) to the horizontal coordinate system based on the acquisition time of each point. Next, the position (position of the camera 102) in the fourth coordinate system (ground coordinate system) of the origin of the second coordinate system (camera coordinate system) at the acquisition time of the image acquired by the camera 102 and the time are the positions of the sun. Is input to the data table or program to obtain the position of the sun on the celestial sphere at that time, that is, the direction in which the sunlight falls in the fourth coordinate system (ground coordinate system).

Next, in the point cloud data in the fourth coordinate system (ground coordinate system), a point-like portion where the road surface (ground surface) is not exposed to sunlight and a shadow is formed is calculated. This process is executed by projecting the point cloud acquired by the laser scanner 101 onto the ground with the sun as the projection origin. For example, if there is a point cloud between the optical axis of the sun's rays and a certain point on the ground surface, the point on the ground surface becomes a shadow portion. This collection of shadow points is a shadow area point cloud. By performing this process on all the point clouds obtained by the point cloud data acquisition unit 202, the area to be the shadow of the ground surface can be determined. FIG. 5 shows an example of a point cloud acquired by a laser scanner by a laser scan and a shadow region point cloud obtained by projecting this point cloud on a road surface.

The shadow area point cloud projection unit 205 projects the shadow area point cloud calculated by the shadow area point cloud calculation unit 204 onto the image captured by the camera 102 by using the result of the coordinate conversion performed by the coordinate conversion unit 203. By this processing, it becomes possible to visually grasp how the point cloud measured by the laser scanner 101 and the shadow region point cloud obtained in the horizontal coordinate system look in the image captured by the camera 102.

Hereinafter, an example of the processing performed by the shadow region point cloud projection unit 205 will be described. First, from the result of the coordinate transformation between the first to fourth coordinate systems described above, the direction and angle of view (photographing field of view) of the optical axis of the camera 102 are described on the fourth coordinate system (horizontal coordinate system). Is possible. On the other hand, the shadow region point cloud described above can be described on the fourth coordinate system (horizontal coordinate system). Therefore, the position of the shadow area point cloud in the shooting range (shooting screen) of the camera 102 can be specified. That is, the shadow area point cloud can be projected onto the shooting range (shooting screen) of the camera 102, and the shadow area point cloud can be displayed in the shooting range.

By projecting a shadow region point cloud into an image captured by the camera 102, image information regarding which portion based on the point cloud data becomes a shadow is displayed in the image captured by the camera 102. FIG. 6 shows an image of a camera image in which a shadow region point cloud is projected. FIG. 6 shows a state in which a group of shadow region points constituting a shadowed portion of the building is projected in an image of the road surface. In the case of FIG. 6, the sun is in the upper left direction of the screen, and it is shown that a shadow region point cloud is formed in the region corresponding to the shadow generated by the sunlight being blocked by the building. The shadow area point cloud is based on the point cloud data of the building obtained by laser scanning and the calculated position of the sun, and it does not matter how the shadow is actually reflected in the image. Therefore, regardless of how the shadow of the building is reflected on the screen, the position of the shadow on the shooting screen (camera image) is specified by the shadow area point cloud.

In the image (camera image) captured by the camera 102, the shadow area expansion unit 206 expands the shadow area around the shadow area point cloud based on the brightness values of the pixels constituting the image. Generally, the point density of a point cloud is coarser than that of pixels in an image. Since the shadow area point cloud is a point cloud composed of point-like shadows, the density is coarser than that of the pixels of the image. Therefore, in order to determine the boundary portion between the shadow and the non-shadow portion with high accuracy, it is necessary to extend the shadow region to the adjacent region of the shadow region point cloud. This process is performed by the shadow region expansion unit 206.

Hereinafter, an example of the processing performed by the shadow area expansion unit 206 will be described. First, a point at the edge of the shadow region point cloud (this point is the point of interest) is extracted. Next, the brightness values of the pixels constituting the image are examined from this point toward the surroundings (along the radial direction). The range to be examined is 360 ° in circumference, which is about twice the distance to the adjacent point of interest.

Here, the gradient of the change in the luminance value of a plurality of pixels arranged in a line is examined. The position where this gradient exceeds the threshold value is acquired as the boundary point of the shadow. For example, when the gradient of the brightness values of the three pixels arranged in a straight line is examined, the central pixel portion of the set of the three pixels having the maximum gradient is acquired as the boundary point. This boundary point is acquired at one or more points of interest for one point of interest, and further at all points of interest of interest. Then, the contour of the shadow region is determined by connecting the obtained plurality of boundary points. The inside of the outline of the shadow area is filled as a shadow.

The shadow specifying unit 207 identifies a shadow portion (a portion of the ground surface (road surface) that is not exposed to sunlight) defined by the process of the shadow region expansion portion 206. The contrast adjusting unit 208 adjusts the contrast of the shadow portion of the road surface. Specifically, the contrast is adjusted so that the display emphasizes the contrast on the premise that the image is a shadow and a dark image. By performing contrast processing on the premise that the image is a shadow portion and the image is dark, it is possible to improve the detection accuracy of the cracked portion of the road surface in the shadow portion.

The shadow boundary line specifying unit 209 specifies the edge of the shadow portion (the boundary between the shadow and the non-shadow portion) defined by the processing in the shadow region expansion portion 206. The edge portion of the shadow is linear, and this linear portion may be erroneously detected as a cracked portion of the road surface. By identifying in advance the edge portion of the linear shadow that may be erroneously detected, the reliability of the finally obtained data can be improved.

The crack portion detection unit 210 detects a crack portion on the road surface from the image captured by the camera 102 by image analysis. This process is performed using ordinary image recognition technology. As this technique, the technique described in Patent Document 1 or the like described in the column of "conventional technique" can be used.

The false detection determination unit 211 determines the correctness of the cracked portion detected by the cracked portion detecting unit 210 based on the shadow boundary line (shadow edge) obtained by the shadow boundary line specifying unit 209. In this process, the position of the shadow boundary line on the shooting screen is compared with the position of the cracked portion detected by the cracked portion detection unit 210 on the shooting screen, and it can be considered that they match or match. (When it is recognized that the match is within the error range), it is determined that the detection of the crack portion detection unit 210 is incorrect. By this processing, the reliability of the inspection data can be improved.

The overhead line detection unit 212 detects an overhead line installed in the sky. Examples of overhead lines include electric wires, telephone lines, optical fiber cables, cable TV signal lines, branch lines (stay lines) of utility poles, and the like. Although it depends on the scan density of the laser scanner 101, a part of the overhead line may be acquired as point cloud data. Since the overhead line is in the sky, there are no point clouds such as other buildings in close proximity, and it extends in an isolated line. Therefore, when two or more point clouds separated from each other are detected in an isolated space, they are connected and specified as an overhead line. Whether or not it exists in isolation and whether or not it is separated from a building or the like is determined using a threshold value. Further, the data of the image obtained by taking an image of the sky can be used to identify the point cloud data presumed to be an overhead line obtained from the above-mentioned laser scan data. However, in this case as well, a part of the overhead line must be detected.

The overhead line detection unit 212 obtains the data of the position of the overhead line by calculation on an appropriate coordinate system such as ground coordinates. For example, when a point cloud of two isolated points is obtained, an overhead line connecting the two points is set, an equation of the straight line or a curve is obtained, and the coordinates are acquired. That is, the overhead line is linear and may not be exposed to the scan light, and the point cloud may be missed. Alternatively, there are cases where point cloud data can be obtained only for a part of the overhead line. In this case, paying attention to the fact that the overhead line extends linearly, the point cloud of the missing part is predicted. For example, by preparing multiple linear equations and curve equations that draw a gentle curve and substituting the partially obtained overhead line point clouds into them, the line equations that fit the obtained point clouds are found. , The position of the overhead line is specified by the equation of the line. The data of the position of the overhead line is used in the calculation performed by the shadow area point cloud calculation unit 204, and the shadow area point cloud corresponding to the overhead line is obtained. That is, a shadow region point cloud corresponding to the shadow generated by the presence of the overhead line can be obtained.

The shadow of the overhead line is linear, and when projected on the road surface, it is easily detected as a crack on the road surface (it is often difficult to identify it by image analysis processing). By obtaining the shadow region point cloud caused by the overhead line, it is possible to reduce the false detection caused by the overhead line and improve the reliability of the data related to the crack of the obtained road surface.

The storage unit 213 stores data handled by the image processing device 200 and data of a program that operates the image processing device 200. In addition, basic data and the like required for calculation are stored in the storage unit 213.

The vehicle position / attitude calculation unit 214 calculates the position and attitude of the vehicle 100 at the designated time t. Since the external orientation elements (position and orientation) of the laser scanner 101 and the camera 102 with respect to the vehicle 100 are known, if the position and orientation of the vehicle at time t can be obtained, the position and orientation of the laser scanner 101 and camera 102 at time t can be obtained. Also sought.

The locus of movement of the vehicle 100 is recorded every moment by GNSS data and IMU data. From this data, the position and posture of the vehicle 100 at the designated time t can be obtained by calculation. This process is performed by the vehicle position specifying unit 214. By specifying the position of the vehicle 100 at time t, the position of the sun at time t, the position of the origin of the scanner coordinate system, and the position of the origin of the camera coordinate system are determined. In addition, the orientation of each coordinate system is determined by finding the posture.

(Example of operation)
FIG. 4 shows an example of the processing procedure executed by the image processing apparatus 200. The program that executes the process of FIG. 4 is stored in the storage unit 213 or an appropriate storage medium, read from the storage unit 213, and executed. The program can also be obtained from a server or a remote storage device via an appropriate line.

In the process of FIG. 4, the processes of steps S101 to S102 are performed on the vehicle 100 side, and the processes of steps S103 to S112 are performed on the image processing device 200. First, while the vehicle is running, the camera 102 is used to photograph the road surface at specific time intervals (for example, 0.1 second intervals). At this time, a laser scan is performed by the laser scanner 101 in synchronization with the shooting of the camera 102 to obtain point cloud data of the surrounding environment. In laser scanning, the scan range is set to scan buildings and overhead lines that may cast shadows on the road surface. The image data and the point cloud data, the time at which the data was obtained, and the posture and position of the vehicle at that time (the posture and position of the IMU 103) are associated and stored in the storage device 105 (steps S101 and S102).

After obtaining a series of data, the data is transferred from the storage device 105 to the image processing device 200, and the processing of step S103 or less is executed. First, to obtain the position and orientation of the laser scanner 101 and the camera 102 at time _{t 1} (step S103). Here, the position and posture data are calculated by the vehicle position / posture calculation unit 214. This data is used when converting the scanner coordinate system and the camera coordinate system into the IMU coordinate system. Then, to get the position of the sun at the time _{t 1} (step S104), and calculates the shadow area point group at time _{t 1} (step S105). At this time, the shadow area point cloud is also calculated for the shadow of the overhead line.

After obtaining the shadow region point group is projected on the camera image captured it at time t ₁ (step S106). Next, extend the shadow area point group on the camera image at time t ₁ (step S107), and determine the area of the shadow in the relevant camera image (step S108).

Next, an adjustment is made to emphasize the contrast of the image portion determined as the shadow region (step S109). By adjusting this contrast, the accuracy of detecting cracks from a shadowed and dark image is improved. After that, an image analysis process for detecting the cracked portion of the road surface is performed (step S110), and the result is compared with the information of the shadow region obtained in step S108 to perform correctness determination (step S111). By making a correct / incorrect judgment, the inconvenience of recognizing the edge of the shadow as a crack can be avoided.

Above processing time _{t 2, the} time _{t 3,} repeated at time _t 4 · · · ·, performed on all of the camera images of interest (step S112).

It is also possible to perform the processing of FIG. 4 in real time. In this case, the image processing device 200 is placed on the vehicle 100, and the processing of FIG. 4 is performed while the vehicle is running.

(Superiority)
In the technique described above, the area of the road surface where the sunlight does not cast a shadow on the building or the like is calculated by using the point cloud data and the position of the sun. In image analysis, even if it is difficult to determine whether it is a shadow edge or a crack on the road surface, if the shadow part is known in advance, the problem of recognizing the shadow edge part as a cracked part by the correctness judgment process. Is avoided. Further, the shadow part is dark and the crack detection accuracy is lowered, but if the shadow part is obtained in advance from the point cloud data, the crack detection accuracy can be improved by adjusting the contrast of the part. ..

In addition, the shadow of an overhead line such as an electric wire is reflected on the road surface, which causes a problem of erroneously detecting the shadow of the overhead line as a crack. However, if the information on the shadow of the overhead line is obtained in advance from the point group data By eliminating the data in that part, it is possible to avoid the problem of recognizing the shadow of the overhead line as a crack.

(Other)
The present invention is not limited to the case of detecting cracks on the road surface, and can also be used for detecting cracks on slopes and cliffs hardened with concrete or mortar, detecting cracks on piers of bridges, and the like.

Claims (8)

An image data acquisition unit that acquires image data of an image of an object taken by a camera,
A point cloud data acquisition unit that acquires point cloud data acquired by a laser scanner,
A shadow area point cloud calculation unit that calculates a shadow area point cloud that projects the point cloud data from the sun onto an object based on the position of the sun.
A shadow region point cloud projection unit that projects the shadow region point cloud onto the image,
With an overhead line detection unit that detects overhead lines based on the point cloud data
With
The shadow area point cloud calculation unit calculates a point cloud of a portion to be a shadow of the detected overhead line, and calculates the point cloud.
An image processing apparatus characterized in that the overhead line detection unit performs a process of predicting a position where the overhead lines are continuously present based on a point cloud of the overhead lines . A coordinate conversion unit for converting the point cloud data acquired by the laser scanner into the ground coordinate system and converting the point cloud data described in the ground coordinate system into the camera coordinate system is further provided.
The image processing apparatus according to claim 1, wherein the calculation of the shadow region point cloud is performed in the horizontal coordinate system. The image processing apparatus according to claim 1 or 2, wherein the image includes a shadow area expansion portion that expands a shadow area around the shadow area point cloud based on a difference in pixel brightness value depending on a position. The image processing apparatus according to claim 3 , further comprising a contrast adjusting unit for adjusting the contrast of the shadow region projected on the image in the image. The image processing apparatus according to claim 3, further comprising a shadow boundary line specifying portion that specifies a shadow boundary line in the image based on the expanded shadow region. A crack portion detection unit that detects a crack portion to be photographed from the image, and a crack portion detection unit.
The image processing apparatus according to claim 5, further comprising a false detection determination unit that determines false detection of a crack portion detected by the crack portion detection unit based on the position of the boundary portion in the image. Image data acquisition step to acquire image data of the image of the object taken by the camera,
The point cloud data acquisition step for acquiring the point cloud data acquired by the laser scanner, and
A shadow region point cloud calculation step for calculating a shadow region point cloud obtained by projecting the point cloud data from the sun onto the object based on the position of the sun.
A shadow region point cloud projection step for projecting the shadow region point cloud onto the image,
With the overhead line detection step of detecting the overhead line based on the point cloud data
Have,
In the shadow area point cloud calculation step, the point cloud of the portion to be the shadow of the detected overhead line is calculated.
An image processing method characterized in that, in the overhead line detection step, a process of predicting a position where the overhead lines are continuously present is performed based on a point cloud of the overhead lines . A program that is read and executed by a computer
An image data acquisition unit that acquires image data of an image of an object taken by a computer with a camera,
A point cloud data acquisition unit that acquires point cloud data acquired by a laser scanner,
A shadow area point cloud calculation unit that calculates a shadow area point cloud by projecting the point cloud data from the sun onto the object based on the position of the sun.
A shadow region point cloud projection unit that projects the shadow region point cloud onto the image,
It functions as an overhead line detection unit that detects overhead lines based on the point cloud data.
The shadow area point cloud calculation unit calculates a point cloud of a portion to be a shadow of the detected overhead line, and calculates the point cloud.
An image processing program characterized in that the overhead line detection unit performs a process of predicting a position where the overhead lines are continuously present based on a point cloud of the overhead lines .

Images