JP7729761B2 - Determination program, determination method, and information processing device - Google Patents

Description

The present invention relates to a determination program, a determination method, and an information processing device.

Public roads are inspected regularly to ensure safe passage. Road inspections include not only the condition of the road surface, but also, for example, the condition of the slopes on the side of the road. If any abnormalities or signs of problems are discovered during the inspection, repair work is carried out in the affected areas.

Inspections of roads and their surrounding areas are typically carried out visually by humans. However, it is difficult to carry out visual inspections frequently for each of the vast number of roads. Furthermore, visual inspections are prone to inconsistencies in judgment between inspectors, making the quality of the inspections unstable.

Considering this, one approach is to use images captured by cameras for inspection. For example, technology has been proposed that uses images captured by a camera capturing the area around a moving object to detect changes in the path of the moving object or in structures surrounding the path.

International Publication No. 2018/155149

For example, images taken with a camera can be subjected to segmentation, a machine learning technique, to classify the types of objects captured in the image. For images taken around a road, segmentation can be used to determine whether the object in the image is a road or a rock. Segmentation can be used to detect falling rocks on the road, for example.

However, it can be difficult to determine whether an image is abnormal or not using image segmentation alone. For example, if the slope beside the road is a stone wall, image segmentation will classify each stone stacked in the stone wall as a stone. As a result, even if there are stones in the area adjacent to the road, it is not possible to determine whether the stone is a fallen rock or a stone that is part of the stone wall.

In one aspect, the present invention aims to improve the accuracy of anomaly detection using images.

In one proposal, a determination program is provided that causes a computer to execute the following processes.
The computer identifies the slope area occupied by the slope and the road area occupied by the road when observing the predetermined area from a predetermined position based on measurement data that measures the positions of the slope and road surface in the predetermined area.The computer detects a partial area in which an object that may be abnormal is captured based on fixed-point observation data that includes images of the predetermined area captured from the predetermined position.The computer then determines whether an abnormality has occurred in the predetermined area based on whether the partial area is included in the slope area or the road surface area.

According to one aspect, the accuracy of anomaly detection using images is improved.

FIG. 10 illustrates an example of a determination method according to the first embodiment. FIG. 1 is a diagram illustrating an example of a system configuration for inspecting roads. FIG. 2 is a diagram illustrating an example of hardware of a road inspection server. FIG. 2 is a block diagram showing the functions of a road inspection server. FIG. 2 illustrates an example of an image data storage unit. FIG. 2 illustrates an example of a point cloud data storage unit. 10 is a flowchart showing an example of a procedure for road inspection processing. FIG. 10 is a diagram illustrating an example of an image data acquisition method. FIG. 10 is a diagram illustrating an example of a method for generating fixed point observation data. FIG. 10 is a diagram illustrating an example of a category classification result. FIG. 10 is a diagram illustrating an example of a procedure for category classification processing. FIG. 10 is a diagram showing an example of segmentation of image data showing a stone wall. FIG. 10 is a diagram illustrating an example of segmentation of point cloud data around a stone wall. FIG. 10 is a diagram showing an example of rockfall detection using the segmentation results. 10A and 10B are diagrams illustrating an example of processing for indicating a fallen rock portion in image data. FIG. 10 is a diagram illustrating an example of correction processing for a front-facing image. FIG. 10 is a diagram illustrating an example of a depth map creation process. FIG. 10 is a diagram illustrating an example of abnormality detection by differential extraction. FIG. 10 is a diagram showing an example of abnormality detection based on the speed of crack progression. FIG. 10 is a diagram showing an example of an abnormality display screen.

The present embodiment will be described below with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments within a range that does not contradict each other.
First Embodiment
First, a first embodiment will be described. The first embodiment is an abnormality determination method that can detect abnormalities around a road with high accuracy based on images of a slope and a road surface and the positions of the slope and the road surface measured by a LiDAR (Light Detection and Ranging) sensor.

Figure 1 shows an example of a determination method according to the first embodiment. Figure 1 also shows an information processing device 10 that implements the determination method. The information processing device 10 can implement the determination method described below, for example, by executing a determination program.

The information processing device 10 has a memory unit 11 and a processing unit 12. The memory unit 11 is, for example, a memory or storage device included in the information processing device 10. The processing unit 12 is, for example, a processor or arithmetic circuit included in the information processing device 10.

The information processing device 10 acquires image data 2a, 2b, ... and measurement data 3a, 3b, ... from, for example, a camera 1a and a LiDAR sensor 1b mounted on a vehicle 1. The vehicle 1 is, for example, a route bus that regularly passes through a road including a predetermined area to be surveyed. The camera 1a captures images of the exterior of the vehicle while the vehicle 1 is traveling, and generates image data 2a, 2b, .... The image data 2a, 2b, ... are assigned date and time information indicating the date and time the image was taken, and location information indicating the location where the image was taken. The location information is, for example, latitude and longitude information measured by a GPS (Global Positioning System). Note that by using GPS in combination with a quasi-zenith satellite system, the location where the image was taken can be measured with high accuracy.

Measurement data 3a, 3b, ... is data obtained by measuring the positions of slopes and road surfaces in a specified area. For example, LiDAR sensor 1b irradiates a measurement object with laser light and measures the distance and direction to the measurement object based on the reflected light. Based on the measured distance and direction, the three-dimensional coordinates of the point irradiated with the laser light are obtained. These coordinates are relative to LiDAR sensor 1b. LiDAR sensor 1b obtains the coordinates of multiple points on the measurement object by irradiating laser light in different directions around it. LiDAR sensor 1b then generates point cloud data indicating the coordinates of each measured point as measurement data 3a, 3b, .... Measurement data 3a, 3b, ... is provided with date and time information indicating the date and time of measurement and location information indicating the measurement location. The absolute position (latitude and longitude) of the measured point can be determined based on the location information indicating the measurement location and the coordinates shown in the point cloud data.

The information processing device 10 stores the acquired image data 2a, 2b, ... and measurement data 3a, 3b, ... in the memory unit 11. The processing unit 12 determines the presence or absence of abnormalities around the road based on the image data 2a, 2b, ... and measurement data 3a, 3b, ... stored in the memory unit 11.

For example, based on the measurement data 3a, 3b, ..., the processing unit 12 identifies the slope area 5a occupied by the slope and the road surface area 5b occupied by the road surface when observing a predetermined area from a predetermined position. The predetermined position is, for example, the position of the camera 1a when the image used to determine the presence or absence of an abnormality (the image included in the fixed-point observation data 4) was captured. The predetermined area is, for example, an area within the range captured in the image used to determine the presence or absence of an abnormality. In other words, the processing unit 12 identifies the slope area 5a and the road surface area 5b from the area that can be observed within the angle of view of the camera 1a in the optical axis direction of the camera 1a, from the position of the camera 1a when the image used to determine the presence or absence of an abnormality was captured.

The slope area 5a and the road surface area 5b can be identified, for example, by segmenting the point cloud data. For example, the processing unit 12 has a trained model for segmentation based on the point cloud data, generated by machine learning. The processing unit 12 inputs the latest point cloud data into the trained model to identify the slope area 5a and the road surface area 5b.

The processing unit 12 also detects partial areas containing potentially abnormal objects based on fixed-point observation data 4, which includes images of a specified area photographed from a specified position. The fixed-point observation data 4 is created based on multiple image data 2a, 2b, .... For example, the processing unit 12 extracts multiple time-series image data (second image data) of a specified area photographed at different dates and times from the image data 2a, 2b, ... (first image data) stored in the memory unit 11, based on the location information. The processing unit 12 then generates fixed-point observation data 4 containing the extracted multiple second image data.

By detecting partial regions based on the fixed-point observation data 4, regions containing, for example, stones 6a, soil 6b, cracks 6c, etc. are detected as partial regions. Detection of partial regions can be achieved by segmenting the image shown in the fixed-point observation data 4. For example, the processing unit 12 has a trained model for segmentation based on image data, generated by machine learning. The processing unit 12 inputs the latest image data into the trained model to identify the category of objects contained in the image. In this case, the processing unit 12 detects regions containing objects belonging to a predetermined category (stones 6a, soil 6b, cracks 6c, etc.) as partial regions.

The processing unit 12 can also detect, as a partial region, an area where there is a difference between multiple images of a specified area in the fixed-point observation data 4 taken at different dates and times. For example, the processing unit 12 extracts the difference between the most recent image data of the specified area and image data taken one month ago. If a rockfall occurred within the past month, the area in which the stone 6a appears is extracted as the difference. Similarly, if soil 6b has accumulated on a slope within the past month, the area in which the soil 6b appears is extracted as the difference.

The processing unit 12 then determines whether an abnormality has occurred in a specified region based on whether the partial region is included in the slope region 5a or the road surface region 5b. For example, assume that an area containing a stone 6a is detected as a partial region. In this case, if the partial region is within the road surface region 5b, the processing unit 12 determines that the partial region has an abnormality (type: "falling rock"). Also assume that an area containing a crack 6c is detected as a partial region. In this case, if the partial region is within the slope region 5a, the processing unit 12 determines that the partial region has an abnormality (type: "crack in slope").

The processing unit 12 may detect areas in each of the multiple images that contain cracks as partial areas, and when it determines that a crack is present on the slope, it may determine the level of risk based on the size and progression speed of the crack along with that determination. For example, the processing unit 12 may increase the level of risk the faster the crack progression speed.

In this way, it is possible to accurately determine whether an abnormality has occurred based on whether the partial area containing a potentially abnormal object is within the slope area 5a or the road surface area 5b. For example, even if a stone is captured in the image, if the partial area containing the stone is within the slope area 5a, the stone is considered to be part of a stone wall and is not abnormal. On the other hand, if the partial area containing the stone is within the road surface area 5b, it can be correctly determined that an abnormality such as a falling rock has occurred.

Furthermore, by determining the slope area 5a and road surface area 5b through segmentation of the point cloud data acquired as measurement data 3a, 3b, ..., the slope area 5a and road surface area 5b can be determined with high accuracy. Improving the accuracy of determining the slope area 5a and road surface area 5b also improves the accuracy of abnormality detection.

Furthermore, by using image segmentation to identify the category of objects in the image and detecting areas containing objects belonging to a specific category as partial regions, it is possible to extract areas containing objects of any category. This makes it easier to detect falling rocks, cracks, etc.

Furthermore, by detecting areas where there are differences between multiple images taken at different dates and times as partial regions, it is possible to prevent overlooking abnormalities. For example, if something that should be there is missing, such as a fallen guardrail, it cannot be detected using the most recent image alone. By detecting differences with previous images, it is also possible to detect abnormalities such as a fallen guardrail. Furthermore, by treating areas where there are differences between multiple images as partial regions, it is possible to extract areas that are difficult to distinguish using segmentation, such as cracks in slopes, as partial regions. As a result, it is possible to prevent overlooking abnormalities.

By detecting areas where there are differences between multiple images taken at different times and dates as partial areas and determining that there is a crack on the slope, the risk level can be determined based on the size and rate of progression of the crack, making it possible to accurately determine the risk level of the crack on the slope.

Furthermore, by detecting abnormalities based on images captured by a camera 1a mounted on a vehicle 1 such as a route bus, it is possible to quickly detect any abnormalities that may occur. For example, by transmitting image data representing images captured by the camera 1a to the information processing device 10 in real time via wireless communication, it becomes possible to quickly discover and address abnormalities in the road surface or slope. As a result, safety around roads can be improved.

Second Embodiment
Next, a second embodiment will be described. The second embodiment is a system that captures images of road conditions using a camera mounted on a vehicle that periodically patrols a road to be monitored, and detects road abnormalities at an early stage based on the captured images.

Figure 2 shows an example of a system configuration for inspecting roads. A road inspection server 100, a GIS (Geographic Information System) server 200, and a weather information server 300 are connected to a network 20. A wireless communication base station 20a is also provided on the network 20.

The road inspection server 100 is a computer that detects abnormalities in roads, slopes, etc. based on information such as images showing the condition of the road. The GIS server 200 is a computer that uses geographical locations as clues to manage, process, and otherwise manage data containing information about those locations. The weather information server 300 is a computer that distributes weather information.

Vehicles 30, 40 are used to collect information indicating road conditions. Vehicle 30 is a vehicle that patrols the monitored roads at a high frequency and on a regular basis, such as a route bus, school bus, or postal vehicle. Vehicle 30 is equipped with a camera 31 that can wirelessly transmit data to road inspection server 100 via base station 20a. Camera 31 captures images of the road and its surroundings while vehicle 30 is traveling, and transmits the captured images to road inspection server 100. When primarily inspecting road slopes, camera 31 is installed facing sideways relative to the direction of travel of vehicle 30 (for example, 90 degrees off the direction of travel).

Vehicle 40 is a vehicle used by road inspection workers for inspections. Vehicle 40 is equipped with a LiDAR sensor 41 that can wirelessly transmit data to road inspection server 100 via base station 20a. Vehicle 40 travels on the road to be inspected. While vehicle 40 is traveling, LiDAR sensor 41 measures the distance from vehicle 40 to points on the road and surrounding objects, as well as the direction of those points. For each measured point on the object, LiDAR sensor 41 calculates coordinates indicating the position of that point and generates point cloud data indicating the coordinates of each of the measured points. LiDAR sensor 41 transmits the generated point cloud data to road inspection server 100. When primarily inspecting road slopes, LiDAR sensor 41 is installed sideways relative to the vehicle's 40's direction of travel (for example, 90 degrees off the direction of travel).

Figure 3 is a diagram showing an example of the hardware of a road inspection server. The road inspection server 100 is controlled entirely by a processor 101. A memory 102 and multiple peripheral devices are connected to the processor 101 via a bus 109. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). At least some of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device).

Memory 102 is used as the main storage device of the road inspection server 100. Memory 102 temporarily stores at least a portion of the OS (Operating System) programs and application programs executed by processor 101. Memory 102 also stores various data used in processing by processor 101. A volatile semiconductor storage device such as RAM (Random Access Memory) is used as memory 102, for example.

Peripheral devices connected to the bus 109 include a storage device 103, a GPU (Graphics Processing Unit) 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.

The storage device 103 writes and reads data electrically or magnetically to the built-in recording medium. The storage device 103 is used as a secondary storage device for the road inspection server 100. The storage device 103 stores the OS program, application programs, and various data. Note that the storage device 103 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

The GPU 104 is a computing device that performs image processing and is also called a graphics controller. A monitor 21 is connected to the GPU 104. The GPU 104 displays images on the screen of the monitor 21 in accordance with instructions from the processor 101. The monitor 21 may be a display device that uses organic EL (Electro Luminescence) or a liquid crystal display device.

The input interface 105 is connected to a keyboard 22 and a mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and mouse 23 to the processor 101. Note that the mouse 23 is an example of a pointing device, and other pointing devices can also be used. Other pointing devices include a touch panel, tablet, touchpad, trackball, etc.

The optical drive device 106 uses laser light or other light sources to read or write data recorded on an optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflected light. Optical discs 24 include DVDs (Digital Versatile Discs), DVD-RAMs, CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable)/RWs (Rewritable).

The device connection interface 107 is a communications interface for connecting peripheral devices to the road inspection server 100. For example, a memory device 25 or a memory reader/writer 26 can be connected to the device connection interface 107. The memory device 25 is a recording medium equipped with a communication function with the device connection interface 107. The memory reader/writer 26 is a device for writing data to the memory card 27 or reading data from the memory card 27. The memory card 27 is a card-type recording medium.

The network interface 108 is connected to the network 20. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 108 is a wired communication interface that is connected by a cable to a wired communication device such as a switch or router. The network interface 108 may also be a wireless communication interface that is connected by radio waves to a wireless communication device such as a base station or access point.

The road inspection server 100 can be realized using the hardware described above. The GIS server 200 and weather information server 300 can also be realized using hardware similar to that of the road inspection server 100. The information processing device 10 shown in the first embodiment can also be realized using hardware similar to that of the road inspection server 100 shown in Figure 3.

The road inspection server 100 realizes the processing functions of the second embodiment by executing a program recorded on, for example, a computer-readable recording medium. The program describing the processing to be executed by the road inspection server 100 can be recorded on various recording media. For example, the program to be executed by the road inspection server 100 can be stored in the storage device 103. The processor 101 loads at least a portion of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the road inspection server 100 can also be recorded on a portable recording medium such as the optical disc 24, memory device 25, or memory card 27. A program stored on a portable recording medium becomes executable after being installed on the storage device 103, for example, under the control of the processor 101. The processor 101 can also read and execute the program directly from the portable recording medium.

In the system shown in Figure 2, vehicle 30 sends images of the road to road inspection server 100 at high frequency (for example, several times a day). The sent images are accompanied by location information obtained by a positioning system such as GPS. Vehicle 40 also sends point cloud data to road inspection server 100 at regular intervals. The sent point cloud data is accompanied by location information obtained by a positioning system such as GPS. Road inspection server 100 uses images and point cloud data of the same location to determine whether there are any abnormalities on the road and its surrounding area.

Figure 4 is a block diagram showing the functions of the road inspection server. The road inspection server 100 has an image data receiving unit 110, an image data storage unit 120, a point cloud data receiving unit 130, a point cloud data storage unit 140, an image preprocessing unit 150, a segmentation unit 160, a difference detection unit 170, and a visualization unit 180.

The image data receiving unit 110 receives images sent from the vehicle 30. For example, the vehicle 30 has an image data sending unit 32 that sends image data showing images captured by the camera 31. For example, the image data sending unit 32 adds location information of the vehicle 30 at the time of capture and date and time information showing the date and time of capture to the image data, and sends the image data including the location information and date and time information to the road inspection server 100. The image data receiving unit 110 receives the image data including the location information and date and time information from the image data sending unit 32. The image data receiving unit 110 stores the received image data in the image data storage unit 120.

The image data storage unit 120 stores image data. For example, part of the memory 102 or storage area of the storage device 103 of the road inspection server 100 is used as the image data storage unit 120.

The point cloud data receiving unit 130 receives point cloud data sent from the vehicle 40. For example, the vehicle 40 has a point cloud data transmitting unit 42 that transmits point cloud data indicating the position of each point measured by the LiDAR sensor 41. For example, the point cloud data transmitting unit 42 adds position information of the vehicle 40 at the time of measurement and date and time information indicating the date and time of the image capture to the point cloud data, and transmits the point cloud data including the position information and date and time information to the road inspection server 100. The point cloud data receiving unit 130 receives the point cloud data including the position information and date and time information from the point cloud data transmitting unit 42. The point cloud data receiving unit 130 stores the received point cloud data in the point cloud data storage unit 140.

The point cloud data storage unit 140 stores point cloud data. For example, part of the memory 102 of the road inspection server 100 or part of the storage area of the storage device 103 is used as the point cloud data storage unit 140.

The image preprocessing unit 150 preprocesses the image to make it easier to analyze. For example, the image preprocessing unit 150 performs processes such as normalizing the color information of the image and creating a depth map. The image preprocessing unit 150 can also perform processes such as binarizing the image and extracting contours. The image preprocessing unit 150 sends the preprocessed image to the segmentation unit 160.

The segmentation unit 160 performs segmentation using AI (Artificial Intelligence) technology. For example, the segmentation unit 160 has a trained model for determining the type of object appearing in an image. The model is represented, for example, by a neural network. The segmentation unit 160 inputs the image into the model, performs calculations according to the model, and outputs, as an output result, for each region of the image, the category to which the object appearing in that region belongs.

The segmentation unit 160 also performs segmentation based on the point cloud data of the range captured in the image. For example, the segmentation unit 160 has a trained model for determining the type of object represented by the point cloud data. The model is represented, for example, by a neural network. The segmentation unit 160 inputs the point cloud data into the model, performs calculations according to the model, and outputs as an output whether the object represented by the point cloud data is a slope or a road surface.

The segmentation unit 160 detects problem areas such as falling rocks based on the results of segmenting the image and point cloud data. For example, if a stone of a certain size or larger is present on the road, the segmentation unit 160 determines that the stone is a fallen rock.

The difference detection unit 170 detects the differences between multiple images taken at the same location on different dates and times. When detecting image differences, the difference detection unit 170 uses images that have been preprocessed by, for example, the image preprocessing unit 150. The difference detection unit 170 determines whether or not there is an abnormality based on the differences between images taken on different dates and times. For example, the difference detection unit 170 determines that there is an abnormality if a crack in a slope is rapidly expanding.

When the segmentation unit 160 or the difference detection unit 170 detects an abnormality, the visualization unit 180 outputs information about the abnormal location. For example, the visualization unit 180 outputs an image of the location where the abnormality was detected, along with details of the detected abnormality. The visualization unit 180 also determines the level of danger for the location where the abnormality was detected, and outputs information indicating the level of danger. The level of danger is determined based on, for example, the amount of accumulated soil and sand, the size of the falling rocks, the size and progression speed of the crack, etc.

The visualization unit 180 can also determine the level of risk using information obtainable from the GIS server 200. For example, the GIS server 200 has a slope information DB 210 and a risk assessment DB 220. The slope information DB 210 stores information such as the type of slope for each road location. The risk assessment DB 220 stores information on the risk of the road or slope for each road location. The GIS server 200 can also obtain weather information from the weather information server 300, calculate the risk of the slope based on the most recent weather information, and store the calculated risk in the risk assessment DB 220. For example, the weather information server 300 has a weather information DB 310 that stores information on past weather (such as rainfall). The GIS server 200 obtains weather information for the area around the road being inspected from the weather information DB 310 of the weather information server 300 and reflects this information in the risk level for the relevant location. For example, the GIS server 200 sets a high risk level when an abnormality is detected on the road in a location where the total rainfall over the past few days is above a predetermined value.

Note that the lines connecting the elements shown in Figure 4 represent only part of the communication paths, and communication paths other than those shown in the figure can also be set. Furthermore, the functions of each element shown in Figure 4 can be realized, for example, by having a computer execute a program module corresponding to that element.

For example, each time the road inspection server 100 receives new image data, it determines whether or not there are any abnormalities in the road, slopes, etc. at the location shown in the image data. In doing so, the road inspection server 100 determines whether or not there are any abnormalities not only from the image shown in the image data being determined, but also from point cloud data around the location where the image was taken. The road inspection server 100 also compares the image data with image data showing past images of the same location to determine whether or not there are any abnormalities. Point cloud data for the location where the image was taken can be obtained from the point cloud data storage unit 140, and image data showing past images can be obtained from the image data storage unit 120.

Figure 5 shows an example of an image data storage unit. The image data storage unit 120 stores multiple image data 121, 122, etc. Each image data 121, 122, etc. is assigned the date and time the image was captured and the location coordinates (latitude, longitude) of the capture location.

Figure 6 shows an example of a point cloud data storage unit. The point cloud data storage unit 140 stores multiple point cloud data 141, 142, etc. Each point cloud data 141, 142, etc. is assigned the date and time when the point position was measured and the position coordinates (latitude, longitude) of the LiDAR sensor 41 at the time of measurement. The point cloud data 141, 142, etc. show the three-dimensional relative coordinates of the multiple points whose positions were measured, as viewed from the LiDAR sensor 41.

As shown in Figures 5 and 6, image data 121, 122, ... and point cloud data 141, 142, ... include date and time (date and time information) and location coordinates (location information). Based on these location coordinates, it is possible to identify point cloud data around the location where the image data being determined for the presence or absence of an abnormality was taken. Furthermore, based on the location coordinates, it is possible to identify past image data that was taken in the same location as the image data being determined for the presence or absence of an abnormality.

Next, the procedure of the road inspection process in the road inspection server 100 will be described with reference to a flowchart.
FIG. 7 is a flowchart showing an example of the procedure for road inspection processing. The processing shown in FIG. 7 will be explained below in order of step number. The road inspection server 100, for example, performs the road inspection processing periodically (hourly, daily, etc.). The road inspection server 100 can also perform the road inspection processing in real time every time new image data is received.

[Step S101] The image pre-processing unit 150 selects the latest uninspected image data. For example, the image pre-processing unit 150 selects one piece of uninspected image data from the image data received since the previous road inspection process. Furthermore, when performing road inspection in real time, the image pre-processing unit 150 selects new image data as the image to be inspected when that image data is stored in the image data storage unit 120.

[Step S102] The image pre-processing unit 150 acquires past image data of the same location as the selected image data, and the latest point cloud data of the same location. For example, the image pre-processing unit 150 acquires from the image data storage unit 120 past image data assigned with shooting location coordinates that can be considered the same as the shooting location coordinates of the selected image data within a specified error range. The image pre-processing unit 150 also acquires from the point cloud data storage unit 140 the latest point cloud data assigned with measurement location coordinates that can be considered the same as the shooting location coordinates of the selected image data within a specified error range.

[Step S103] The image pre-processing unit 150 creates fixed-point observation data for the location where the selected image data was captured. For example, the image pre-processing unit 150 arranges the image data captured at the relevant location in chronological order to create fixed-point observation data.

[Step S104] The image preprocessing unit 150 and segmentation unit 160 work together to perform category classification processing on the selected image data. Details of the category classification processing will be described later (see Figure 11).

[Step S105] The segmentation unit 160 determines whether an abnormality has been detected by category classification. For example, if a stone of a predetermined size or larger is present on the road, the segmentation unit 160 determines that an abnormality, a "rockfall," has occurred. If an abnormality is detected, the segmentation unit 160 proceeds to step S109. If no abnormality is detected, the segmentation unit 160 proceeds to step S106.

[Step S106] The image preprocessing unit 150 performs preprocessing on the time-series image data to emphasize feature points. For example, the image preprocessing unit 150 aligns the time-series image data based on feature points. The image preprocessing unit 150 also performs one or more of the following image processing processes on each piece of time-series image data: angle correction, depth map creation, image color information normalization, binarization, and contour extraction, to generate one or more pieces of processed image data per image. This preprocessing reduces the effects of sunlight conditions, shade, vegetation, etc., and emphasizes changes.

[Step S107] The difference detection unit 170 detects differences in images over time based on the fixed-point observation data. For example, the difference detection unit 170 extracts image data at monthly intervals and detects the degree of progression of cracks in the slope each month.

[Step S108] The difference detection unit 170 determines whether or not there is an abnormality based on the image differences. If there is an abnormality, the difference detection unit 170 proceeds to step S109. If there is no abnormality, the difference detection unit 170 proceeds to step S110.

[Step S109] The visualization unit 180 outputs anomaly information indicating the details of the detected anomaly. For example, the visualization unit 180 displays information such as the amount of change in the crack over time, the amount of precipitation at the relevant location, and the alert zone level at the relevant location on the monitor 21.

[Step S110] The image pre-processing unit 150 determines whether to end the road inspection process. For example, the image pre-processing unit 150 determines to end the road inspection process when an operation is performed to instruct the end of the road inspection process. The image pre-processing unit 150 also determines to end the road inspection process when there is no uninspected image data remaining during periodic road inspection processing. If the image pre-processing unit 150 determines that the road inspection process has ended, it ends the process. If the image pre-processing unit 150 determines that the road inspection process should continue, it proceeds to step S101.

Based on the image data acquired in this way, abnormalities on roads and their surrounding areas can be automatically detected. For example, if the vehicle 30 is a route bus, the vehicle 30 will travel along the same roads at approximately the same time every day. If image data of images taken by the camera 31 mounted on the vehicle 30 is sent in real time to the road inspection server 100 and road inspection processing is performed, road abnormalities can be found early.

Figure 8 shows an example of an image data acquisition method. In Figure 8, a route bus route 51a traveled by a route bus vehicle 30 is shown on a map 51. The route bus vehicle 30 travels along the route bus route 51a at a fixed time every day. A camera 31 mounted on the vehicle 30 captures images of the outside of the vehicle while traveling. Image data representing the captured images is then transmitted to the road inspection server 100 by the image data transmission unit 32.

For example, image data 52 of an image taken at survey point 51b is assigned the location information of survey point 51b and the date and time of the image capture. When the road inspection server 100 receives the image data, it can recognize, based on the location information, that the image data 52 contains an image of the area surrounding survey point 51b.

This image data is stored in the image data storage unit 120. The image pre-processing unit 150 can generate fixed-point observation data for specific survey locations based on the image data stored in the image data storage unit 120.

Figure 9 shows an example of a method for generating fixed-point observation data. For example, assume that the road inspection server 100 received image data 53 showing an image of a certain location from the vehicle 30 in July. In this case, the image pre-processing unit 150 acquires, for example, previous image data 54, 55, etc., for the same location as the image data 53, in monthly increments, from the image data storage unit 120. For example, image data 54 is the image data for an image captured one month before (June) the capture date of image data 53, and has location information closest to that of image data 53. Similarly, image data 55 is the image data for an image captured two months before (May) the capture date of image data 53, and has location information closest to that of image data 53.

Even if images are taken on a bus traveling the same route, the areas captured on the screen will differ if the images are taken on different days. Therefore, the image pre-processing unit 150 roughly aligns the other image data 54, 55, etc. based on the coordinates of the shooting location, using, for example, image data 53 as a reference. The image pre-processing unit 150 then performs feature matching between, for example, image data 53 and each of the other image data 54, 55, etc. The image pre-processing unit 150 then performs processes such as enlarging/reducing, shifting the position up/down/left/right, and rotating each of the image data 54, 55, etc. so that characteristic parts of the images overlap. The image pre-processing unit 150 then creates fixed-point observation data 60 by stitching together the multiple processed images 61, 62, 63, etc. in chronological order.

By correcting images using feature matching in this way, it is possible to absorb deviations in the route taken while driving, differences in weather conditions, differences in the angle of view, etc.
The image data to be inspected is then subjected to a segmentation model trained through machine learning to categorize the image regions, determining the category to which objects moving into each region belong.

Figure 10 shows an example of a category classification result. For example, the segmentation unit 160 performs segmentation of image data 52 by inputting the image data 52 into a trained model for segmentation. As a result of the segmentation, the segmentation unit 160 outputs category information 71 indicating the category for each region of the image.

Category information 71 shows the following categories: slopes 71a and 71b, plants 71c, road surfaces 71d, and stones 71e. Each category indicates the category to which objects appearing in the area of image data 52 that overlaps with the area of that category belong when category information 71 is overlaid on image data 52.

The segmentation unit 160 analyzes the category information 71 to determine whether or not an abnormality exists. For example, if there is a large stone 71e on the road surface 71d, the segmentation unit 160 can determine that a rockfall has occurred.

As shown in Figure 10, if a large stone 71e is clearly located in the middle of the road surface 71d, it is possible to determine that this is a fallen rock based solely on the category information generated from image data 52. However, if a stone is located at the edge of the road surface 71d, it is difficult to distinguish whether the stone is a fallen rock or a stone that is part of a stone wall. Therefore, the segmentation unit 160 improves the accuracy of anomaly detection through category classification by also using segmentation based on point cloud data.

11 is a diagram showing an example of a procedure for category classification processing. The processing shown in FIG. 11 will be described below in order of step number.
[Step S201] The segmentation unit 160 performs segmentation based on image data using a trained model for segmenting image data.

[Step S202] The segmentation unit 160 performs segmentation based on the point cloud data obtained from the LiDAR sensor 41 using a trained model for segmenting point cloud data. Segmentation based on point cloud data can clearly distinguish between, for example, road surfaces and slopes. For example, the segmentation unit 160 generates a plane that contains multiple points shown in the point cloud data, and if the generated plane is close to horizontal (the tilt angle is less than a predetermined value), it categorizes the plane as a "road surface." Furthermore, the segmentation unit 160 generates a plane that contains multiple points shown in the point cloud data, and if the generated plane is close to vertical (the tilt angle is greater than a predetermined value), it categorizes the plane as a "slope."

[Step S203] The segmentation unit 160 combines the category information obtained by segmentation based on the image data with the category information obtained by segmentation based on the point cloud data. For example, the segmentation unit 160 calculates the logical product of the area in the category "stone" in the category information obtained by segmentation based on the image data and the area in the category "road surface" in the category information obtained by segmentation based on the point cloud data. This extracts the area containing the image of the falling rock.

In this way, by performing segmentation based not only on image data but also on point cloud data, the accuracy of category classification can be improved. For example, even if the slope is a stone wall, it is possible to clearly distinguish between the stones that make up the wall and fallen rocks.

Figure 12 is a diagram showing an example of segmentation of image data showing a stone wall. When image data 72 showing a stone wall is segmented using an AI-trained model, the categories of objects shown in image data 72 are indicated in category information 73. In the example of Figure 12, category information 73 indicates that the image contains plants 73a, numerous stones 73b that make up the stone wall, road surface 73c, and stones 73d and 73e on road surface 73c.

In this case, the segmentation unit 160 can clearly determine that stones 73d and 73e, which are located on the road surface 73c, are fallen rocks. However, the segmentation unit 160 cannot determine whether the numerous stones 73b that make up the stone wall are fallen rocks on the road surface or part of the stone wall using only category information 73 based on image data 72.

Therefore, the segmentation unit 160 performs segmentation based on the point cloud data acquired by the LiDAR sensor 41.
FIG. 13 shows an example of segmentation of point cloud data around a stone wall. The point cloud shown in point cloud data 74 is a point cloud that exists within a view volume (a space indicating the range of the display object) when the same optical axis and angle of view as those of image data 72 are used. In other words, the point cloud is a large number of points on the surface of the object depicted in image data 72. The point cloud data 74 indicates the position of each point in the point cloud. In the point cloud data 74, the point cloud on the stone wall exists on a plane slightly tilted from the vertical. The point cloud on the road surface exists on a nearly horizontal plane. When segmentation is performed on this point cloud data 74 using a model trained by AI, category information 75 is generated. The category information 75 includes a slope 75a and a road surface 75b.

The segmentation unit 160 detects falling rocks based on category information 73 obtained from the image data 72 and category information 75 obtained from the point cloud data 74 .
FIG. 14 is a diagram showing an example of rockfall detection using the results of segmentation. For example, the segmentation unit 160 binarizes category information 73 obtained from image data 72 into stones and other items. In the example of FIG. 14, binary data 76 is generated in which stones are represented as "1 (white)" and other items are represented as "0 (black)." The segmentation unit 160 also binarizes category information 75 obtained from point cloud data 74 acquired by the LiDAR sensor 41 into road surfaces and other items. In the example of FIG. 14, binary data 77 is generated in which road surfaces are represented as "1 (white)" and other items are represented as "0 (black)."

The segmentation unit 160 performs a pixel-by-pixel logical AND on the two binary data 76, 77. That is, the segmentation unit 160 compares pixels at the same position in the binary data 76, 77, and generates binary data 78 in which only if both are "1 (white)" do they remain "1 (white)" and otherwise are "0 (black)". In the binary data 78, only the fallen rocks are shown in white.

The segmentation unit 160 can clearly show falling rocks in the image data 72 by overlaying the binary data 78 generated by the logical product on the image data 72 .
FIG. 15 is a diagram showing an example of a process for highlighting rockfall portions in image data. For example, the segmentation unit 160 converts the "1 (white)" portion of the binary data 78 into a semi-transparent highlight color (e.g., red) and combines it with the image data 72. In the combined image data 79, fallen rocks 79a and 79b are highlighted. The image data 72 and the point cloud data 74 have the same optical axis and angle of view. This makes it easy to perform the combination shown in FIG. 15.

If the category classification process determines that no abnormalities exist, the difference detection unit 170 uses the image in which the image pre-processing unit 150 has processed the image data to emphasize the characteristic points and determines whether or not there are abnormalities based on changes in the characteristic points over time. For example, the image pre-processing unit 150 calculates the slope of the slope based on the point cloud data acquired from the LiDAR sensor 41, and corrects the image data so that it is aligned directly with the optical axis.

Figure 16 shows an example of correction processing for a front-facing image. For example, if image data 81 shows an inclined slope, the slope before correction appears farther away at the top. Therefore, the rectangular shape of the slope shown in image data 81 is narrower at the top. The image pre-processing unit 150 calculates the slope's angle of inclination relative to the optical axis based on point cloud data acquired by, for example, the LiDAR sensor 41. The image pre-processing unit 150 then performs image angle correction on image data 82 to create a front-facing image. This generates image data 82 in which the slope is directly facing the camera's viewpoint. Correcting the image to a front-facing image eliminates image distortion due to differences in perspective between the images, improving the accuracy of difference detection by the difference detection unit 170.

The image preprocessing unit 150 can also generate a depth map based on the image data. A depth map is data that represents different depths of objects in an image.
17 is a diagram showing an example of a depth map creation process. For example, the image preprocessing unit 150 creates depth maps 84a, 84b, etc. based on multiple pieces of time-series image data 83a, 83b, etc. taken at the same location. By comparing the created depth maps 84a, 84b, etc., areas at different depths become clear. Falling rocks, earth and sand, etc. may be present in areas at different depths.

Figure 18 is a diagram showing an example of anomaly detection using difference extraction. For example, the difference detection unit 170 extracts the difference between image data 85 taken in June and image data 86 taken at the same location in July. In the example of Figure 18, part of the image data 86 shows accumulated soil and stones. The difference data 87 generated by difference extraction shows soil 87a, stones 87b, and cracks 87c.

Comparing the differential data 87 with the category information 75 shown in Figure 13, it can be seen that stone 87b is present on road surface 75b. Therefore, stone 87b is a fallen rock and is detected as an abnormality. On the other hand, soil 87a is located within slope 75a. Soil 87a accumulated in depressions in slope 75a is detected as an abnormality if its amount exceeds a predetermined amount. Crack 87c is located within slope 75a. Crack 87c in slope 75a is detected as an abnormality if its size exceeds a predetermined amount.

By extracting the differences between the time-series image data 85 and 86 in this way, it is possible to easily detect abnormalities such as sediment accumulation, falling rocks, and cracks in the slope. Furthermore, by comparing the time-series image data, the difference detection unit 170 can determine, for example, the speed at which cracks in the slope are progressing.

Figure 19 shows an example of anomaly detection based on the speed of crack progression. For example, image data 91 from May shows a small crack 91a. Image data 92 from June shows a grown crack 92a. Image data 93 from July shows a further grown crack 93a.

The difference detection unit 170 recognizes the degree of progress of the crack by comparing the time-series image data 91-93. The difference detection unit 170 then determines that an abnormality exists if the change in the length of the crack over, for example, one month exceeds a predetermined value. The difference detection unit 170 also determines that an abnormality exists if the length of the crack in the latest image data 93 exceeds a predetermined value.

When an abnormality is detected, the visualization unit 180 works in conjunction with the GIS to calculate the level of risk based on information such as the location information and detection results attached to the image data in which the abnormality was detected. The visualization unit 180 then adds an icon indicating the abnormal location on the map obtained from the GIS server 200. When the abnormal location is clicked, the visualization unit 180 displays various information such as confirmation of the time series data for the abnormal location, changes in differences, and past construction records.

Figure 20 shows an example of an anomaly display screen. When the visualization unit 180 detects an anomaly, it displays, for example, an anomaly detection screen 94 on the monitor 21. The anomaly detection screen 94 has a map display section 94a. The map display section 94a displays a map of the area around the road being inspected, obtained from the GIS server 200. In addition, on the map displayed in the map display section 94a, an icon 94b indicating the detection of an anomaly is displayed at the location where an anomaly has been detected.

When icon 94b is selected, the visualization unit 180 displays a details display section 94c that shows details of the abnormality. The details display section 94c displays, for example, the risk level of the abnormality. The visualization unit 180 determines the risk level based on the abnormality and information obtained from the GIS server 200.

For example, the visualization unit 180 obtains information on the most recent precipitation and soil rainfall index around the abnormal location, and the level of risk due to topographical factors from the GIS server 200. The precipitation and soil rainfall index are information provided by the weather information server 300. The level of risk due to topographical factors can be obtained, for example, from a hazard map managed by the GIS server 200.

The visualization unit 180 then determines the level of risk by taking into account the degree of abnormality obtained from image data, etc. (for example, the depth and width of the crack, and the rate at which the crack is progressing) as well as the most recent amount of precipitation and the alert zone level for the area. For example, the visualization unit 180 increases the risk level the deeper the crack. The visualization unit 180 also increases the risk level the faster the crack is progressing. The visualization unit 180 also increases the risk level the more precipitation there is or the higher the soil rainfall index. Furthermore, the visualization unit 180 increases the risk level the higher the alert zone level.

An image of the location where the abnormality was detected is displayed in the detail display section 94c. If the location is a crack in a slope, an enlarged image of the crack is displayed in the detail display section 94c.
The crack depth can be measured by the LiDAR sensor 41. By orienting the LiDAR sensor 41 90 degrees to the side of the traveling direction of the vehicle 40, the LiDAR sensor 41 can measure the distance to the slope from almost the front. By measuring from the front of the slope, if there is a crack in the slope, the laser light emitted by the LiDAR sensor 41 reaches the depth of the crack. As a result, the distance to the depth of the crack can be measured. For example, the difference detection unit 170 determines the depth of the crack to be the difference between the distance to the deepest part of the part determined to be a crack by analysis of the image data and the distance to its periphery.

The visualization unit 180 may also use distinctive colors and sizes to visually indicate areas that should be repaired as a priority, such as areas at high risk of collapse or areas with a large collapse range, on a map in an easily recognizable manner. For example, the visualization unit 180 may use red to display icons for areas where abnormalities have been detected that have a "high" risk level, and yellow to display icons for areas where abnormalities have been detected that have a "medium" risk level, thereby indicating differences in the level of risk with the color of the icon.

In this way, it is possible to quickly detect abnormalities around roads based on image data and point cloud data acquired by cameras 31 and LiDAR sensors 41 mounted on vehicles 30, 40, such as route buses. In other words, when a vehicle 30 traveling a fixed route, such as a route bus, passes an abnormal location, image data of the abnormal location is sent to the road inspection server 100, and the abnormality is immediately detected. As a result, falling rocks, accumulations of soil and sand due to heavy rain, etc. can be quickly detected, and measures such as removing the obstacle or evacuating residents can be taken promptly.

Other Embodiments
In the second embodiment, the camera 31 and the LiDAR sensor 41 are mounted on separate vehicles 30, 40, but the camera 31 and the LiDAR sensor 41 may also be mounted on a single vehicle.

The above are examples of embodiments, but the configuration of each part shown in the embodiments can be replaced with other parts that have similar functions. Any other components or processes may also be added. Furthermore, any two or more of the configurations (features) of the above-mentioned embodiments may be combined.

REFERENCE SIGNS LIST 1 Vehicle 1a Camera 1b LiDAR sensor 2a, 2b, ... Image data 3a, 3b, ... Measurement data 4 Fixed-point observation data 5a Slope area 5b Road surface area 6a Stone 6b Soil 6c Crack 10 Information processing device 11 Storage unit 12 Processing unit

Claims (10)

based on measurement data obtained by measuring the positions of a slope and a road surface in a predetermined area, a slope area occupied by the slope and a road surface area occupied by the road surface when the predetermined area is observed from a predetermined position;
detecting a partial area in which an object that may be abnormal is captured based on fixed-point observation data including an image of the predetermined area captured from the predetermined position;
determining whether or not an abnormality has occurred in the predetermined area based on whether the partial area is included in the slope area or the road surface area;
A judgment program that causes a computer to execute a process. the measurement data is point cloud data indicating the positions of a plurality of measurement target points on the slope and the road surface within the predetermined area,
In identifying the slope area and the road surface area, the slope area and the road surface area are identified by performing segmentation on the point cloud data using a model that has been trained by machine learning.
The determination program according to claim 1. In detecting the partial region, the image shown in the fixed point observation data is segmented using a model trained by machine learning to identify a category of objects appearing in the image, and an area in which an object belonging to a predetermined category is shown is detected as the partial region.
The determination program according to claim 1 or 2. In the detection of the partial region, a region in which a stone is photographed is detected as the partial region;
In determining whether or not an abnormality has occurred in the predetermined region, if the partial region is within the road surface region, it is determined that a rockfall has occurred in the partial region.
The determination program according to claim 3. In the detection of the partial region, a region in which a crack is captured is detected as the partial region;
In determining whether or not an abnormality has occurred in the predetermined region, if the partial region is within the slope region, it is determined that a crack exists in the slope.
The determination program according to any one of claims 1 to 4. The fixed point observation data shows a plurality of images of the predetermined area taken at different dates and times,
In the detection of the partial region, a region where there is a difference between the plurality of images is detected as the partial region.
The determination program according to any one of claims 1 to 5. In the detection of the partial region, a region in which a crack appears in each of the plurality of images is detected as the partial region;
In determining whether or not an abnormality has occurred in the predetermined region, if the partial region is within the slope region, it is determined that there is a crack in the slope, and a risk level is determined according to the propagation speed of the crack.
The determination program according to claim 6. a plurality of first image data pieces showing images taken at different dates and times by a camera mounted on a vehicle that periodically passes through the predetermined area, the plurality of first image data pieces having date and time information indicating the date and time of the image taking and location information indicating the location of the image taking, extracting from the plurality of first image data pieces a plurality of second image data pieces in time series taken of the predetermined area at different dates and times based on the location information, and generating the fixed-point observation data including the extracted plurality of second image data pieces;
8. The determination program according to claim 6, further causing the computer to execute a process. based on measurement data obtained by measuring the positions of a slope and a road surface in a predetermined area, a slope area occupied by the slope and a road surface area occupied by the road surface when the predetermined area is observed from a predetermined position;
detecting a partial area in which an object that may be abnormal is captured based on fixed-point observation data including an image of the predetermined area captured from the predetermined position;
determining whether or not an abnormality has occurred in the predetermined area based on whether the partial area is included in the slope area or the road surface area;
A method for determining whether a process is executed by a computer. a processing unit that identifies a slope area occupied by the slope and a road area occupied by the road when the predetermined area is observed from a predetermined position based on measurement data that measures the positions of a slope and a road surface in the predetermined area, detects a partial area in which an object that may be abnormal is captured based on fixed-point observation data that includes an image of the predetermined area photographed from the predetermined position, and determines whether or not an abnormality has occurred in the predetermined area based on whether the partial area is included in the slope area or the road surface area;
An information processing device having the above.