JP6995013B2 - Monitoring system - Google Patents

Description

The present invention relates to a technique for observing the dynamics of an object, and more specifically, to a monitoring system for extracting a change region of an object from a single photograph acquired by a fixed-point camera.

It is said that two-thirds of Japan's territory is mountainous, so many people set up their residences on the land behind the slope, and roads and railroad tracks almost always pass by the side of the slope. be. And the slopes have the possibility of disasters such as collapses and landslides, and they have often suffered great damage due to slope failures and the like.

On slopes that are at risk of collapse (including natural and artificial slopes) or slopes with signs of landslides, measurements may be taken to monitor their movement. For example, on slopes with signs of landslides, measurements using extensometers and punching plates, in-hole inclinometers, and ground surface displacement measurements were carried out. However, in the measurement with an extensometer or a punching plate, it is ineffective unless it is installed over the landslide boundary (especially the head), and the in-hole extensometer is also ineffective unless the landslide depth is accurately estimated. There is a problem that it costs money if it is provided in a place.

Ground surface displacement measurement obtains the coordinates of many observation points installed on the slope and monitors the movement of the slope by detecting the displacement over time, so it is possible to directly grasp the abnormality and also to grasp the abnormality. It has the advantage that its effect does not depend on the installation location of the instrument, unlike extensometers and in-hole inclinometers. However, the installation of the observation point is dangerous because people enter the slope, and the person has to position the observation point using a total station or the like, which requires a great deal of labor and cost.

Therefore, various monitoring techniques have been proposed in place of direct measurement by a total station or the like. For example, Patent Document 1 proposes a technique for monitoring the movement of a slope by using an image of the slope.

Japanese Unexamined Patent Publication No. 2016-180681

The technique disclosed in Patent Document 1 generates a three-dimensional model of a slope by stereo matching processing or the like, and calculates the displacement between two periods after obtaining the three-dimensional coordinates of a specific location. Therefore, it is necessary to install cameras at two or more places, and it is necessary to generate a three-dimensional model based on images taken from a plurality of directions. That is, the technique disclosed in Patent Document 1 has a problem that the cost increases by arranging cameras and photographers in a plurality of places, and further, because the image is simply processed mechanically, noise (vegetation shaking) is mistakenly made. It is possible to point out the problem that there is a risk of evaluating (things that are different from the original changes).

An object of the present invention is to solve a conventional problem, that is, a monitoring system capable of monitoring an object after removing predictable noise without having to arrange cameras at a plurality of places. To provide.

The present invention focuses on the point of extracting changes in an object by using an image (single photograph) obtained from a fixed-point camera, and is an invention made based on an idea that has not existed in the past.

The monitoring system of the present invention is a system that monitors changes in an object using an image taken by a fixed-point camera installed at one point, and includes a fixed-point camera, image matching means, candidate area extraction means, and change area extraction means. It is a thing. Of these, the fixed-point camera periodically (or intermittently) acquires images of the object, and the image matching means is a first-stage image (an image acquired by the fixed-point camera in the first period) and a second-stage image. It is a means for aligning the positions of the first period image and the second period image by collating the images (images acquired by the fixed point camera in the second period). Further, the candidate region extraction means is a means for extracting a region relatively moved between the aligned first-stage image and the second-stage image as a "change region candidate", and the change region extraction means is a means for extracting the region. This is a means for extracting a change region candidate whose change amount exceeds a predetermined change amount threshold value as a “change region”.

The monitoring system of the present invention may be further provided with directional noise extraction means. This directional noise extraction means is a means for extracting a change region candidate whose change direction is within a range of a predetermined change direction threshold value as "direction noise". The change direction threshold value is a range set by a predetermined width from the vertical upward direction. In this case, the change region extraction means extracts the change region from the change region candidates excluding the directional noise.

The monitoring system of the present invention may be further provided with time noise extraction means. This time noise extraction means is short with the first period image by collating the first period image and the short time image (one or more images acquired by the fixed point camera between the first period and the second period). It is a means for aligning the position of the time image and extracting a region relatively moved between the aligned first period image and the short-time image as "time noise". In this case, the change region extraction means extracts the change region from the change region candidates excluding the time noise.

The monitoring system of the present invention may be further provided with a search area creating means. In this search area creating means, one of the first period image and the second period image is a "search image" and the other is a "searched image", and the "search area" excluding the peripheral end of the search image. It is a search method to cut out. In this case, the image collation means aligns the positions of the first period image and the second period image by image collating the search area and the searched image.

The monitoring system of the present invention has the following effects.
(1) Since images are continuously acquired and monitored by a fixed-point camera, real-time monitoring and constant monitoring of an object can be realized.
(2) Since it is sufficient to install one fixed-point camera, it is possible to reduce the cost of equipment and the cost of installation as compared with the conventional case.
(3) By removing "direction noise" and "time noise", it is possible to accurately extract a region where there is actually a significant change.

The block diagram which shows the main structure of the monitoring system of this invention. The flow chart which shows the main processing flow of the monitoring system of this invention. The flow diagram which shows the flow of the process which the pair image selection means selects the 2nd period image. A model diagram showing the relationship between the first period image, the short-time image, and the second period image candidate. (A) is an image diagram showing a combination of two images having a small difference value between the first luminance value and the second luminance value, and (b) is an image diagram showing a combination of two images having a large difference value between the first luminance value and the second luminance value. An image diagram showing the combination. (A) is an image diagram showing a search area excluding the peripheral edge of the search image, and (b) is an image diagram showing a searched image to be collated with the search area while changing the position. An image diagram illustrating a target "significant change area" and a "noise area" to be removed. A model diagram showing a change direction threshold set with a predetermined width on both sides from the vertical upward direction. A flow chart showing the flow of processing in which the warning information output means outputs warning information.

An example of an embodiment of the monitoring system of the present invention will be described with reference to the drawings.

First, the outline of the present invention will be described. In the present invention, a region changed in an object by selecting two time periods from a large number of single photographs taken with a fixed point camera and comparing the images of the two time periods (hereinafter referred to as "change area"). ) Is extracted. More specifically, an image obtained by the fixed point camera in the first period (hereinafter referred to as "first period image") and an image obtained by the fixed point camera in the second period (hereinafter referred to as "1st period image"). The change region is extracted by performing pattern matching with the "second period image") and obtaining the positional difference of the partial image (pattern) common between the two images.

Hereinafter, the monitoring system of the present invention will be described in detail. Although various objects can be implemented as the object of the present invention, the case where the object is a slope (natural slope, slope, etc.) will be described here for convenience.

FIG. 1 is a block diagram showing a main configuration of the monitoring system of the present invention. As shown in this figure, the monitoring system 100 of the present invention includes a fixed point camera 101, an image collation means 102, a candidate area extraction means 103, and a change area extraction means 104, and further includes an image storage means 105 and a pair image selection means. It can also include 106, a search area creating means 107, a directional noise extracting means 108, a time noise extracting means 109, and a warning information output means 110.

Among the means constituting the monitoring system 100, the image collation means 102, the candidate area extraction means 103, the change area extraction means 104, the pair image selection means 106, the search area creation means 107, the direction noise extraction means 108, and the time noise extraction means. The 109 and the warning information output means 110 can be manufactured as a dedicated one, or a general-purpose computer device can be used. This computer device can be configured by a personal computer (PC), a tablet PC such as an iPad (registered trademark), a mobile terminal including a smartphone, a PDA (Personal Data Assistance), or the like. The computer device includes a processor such as a CPU and a memory such as a ROM and a RAM, and may further include an input means such as a mouse and a keyboard and a display.

As the fixed point camera 101 constituting the surveillance system 100, a conventionally used camera can be used, and for example, a trail camera for wildlife photography can be used. It is desirable that the fixed-point camera 101 automatically shoots without any human operation, and it is preferable to set the fixed-point camera 101 to shoot periodically (or irregularly or intermittently) at intervals of, for example, 1 hour or 30 minutes. In addition, the fixed-point camera 101 is installed at a position (one point) where a bird's-eye view of the slope, which is an object, can be taken, and since the same direction is taken from the same position, basically the same range of images of the slope is acquired. To. For automatic shooting over a relatively long period of time (several months to a year), a considerable battery is prepared.

Since the fixed-point camera 101 takes pictures for a long period of time and periodically, the number of images to be acquired is enormous. The image storage means 105 stores this image. The image storage means 105 can be constructed on a database server, for example, and can be arranged around the fixed point camera 101 and connected to the fixed point camera 101 (image data communication) by a local network (LAN: LocalAreaNetwork). It can also be a cloud server that saves via the Internet (that is, wireless communication).

FIG. 2 is a flow chart showing the main processing flow of the monitoring system 100 of the present invention. The central column shows the processing to be performed, the left column shows the input information required for the processing, and the right column shows the input information required for the processing. The output information generated from the processing is shown. Hereinafter, the main processing flow of the monitoring system 100 will be described with reference to this figure.

When the fixed point camera 101 acquires the first period image (that is, the first period image is selected from a large number of images), the fixed point camera 101 periodically (intermittently) acquires the first period image after the first period (Step 100). The pair image selection means 106 selects the second period image from the images (Step 200). As described above, the present invention extracts a change region in the slope by comparing the first period image with the second period image (performing pattern matching). However, if the time between the first period and the second period is too short, no significant change can be detected. This is because changes that are not originally intended may be extracted as change areas due to vegetation swaying in the wind, movement of animals such as birds, or sudden changes in light (especially sunlight). Especially in the case of slopes, fluctuations due to collapses and landslides are relatively slow, so it is not appropriate to compare two images with short intervals. Therefore, it was decided to select the second period image from the images acquired after an appropriate time (hereinafter referred to as "predetermined time") has elapsed from the first period when the first period image was acquired. It should be noted that this predetermined time is set in advance, and can be appropriately set according to the situation of the object, for example, 10 minutes, 30 minutes, or 1 hour.

FIG. 3 is a flow chart showing a flow of processing in which the pair image selection means 106 selects a second period image. When the fixed point camera 101 acquires an image after the first period, it is determined whether or not a predetermined time has elapsed from the first period (Step 201). Here, the image acquired without the elapse of the predetermined time is regarded as a "short-time image", while the image acquired after the elapse of the predetermined time is regarded as a candidate for the second period image for the time being. FIG. 4 is a model diagram showing the relationship between the first period image, the short-time image, and the second period image candidate.

The image determined to be a short-time image is stored in the image storage means 105, and the image determined to be a second-stage image candidate is determined to be suitable as a second-stage image (Step 204). Even if the images are acquired at intervals of a predetermined time, proper matching (pattern matching) may not be possible due to changes in sunlight. Therefore, it is advisable to perform pattern matching between the first period image and the second period image candidate, and determine the suitability as the second period image based on the correlation coefficient obtained as a result. That is, when the correlation coefficient exceeds a predetermined threshold value, it is determined that the second period image candidate is suitable as a second period image, and when not, the second period image candidate is used as a second period image. It is judged that there is no suitability for. Alternatively, the luminance value of the first image (hereinafter referred to as "first luminance value") and the luminance value of the second image (hereinafter referred to as "second luminance value") are obtained, and the difference value between the two is predetermined. If it is within a predetermined threshold value (hereinafter referred to as "luminance difference threshold value"), it is determined that the second period image candidate is suitable as a second period image, and if not, the second period image candidate is judged to be suitable. It can also be determined that it is not suitable as a second-stage image. The first luminance value and the second luminance value can be various statistical values such as a total value or an average value of the brightness of all pixels, or a median value or a mode value. In FIG. 5A, a combination of two images (first period image and second period image) having a small difference value between the first brightness value and the second brightness value (suitable as a second period image = good pair). 5 (b) shows a combination of two images having a large difference value between the first luminance value and the second luminance value (not suitable as a second time image = unsuitable pair). As shown in FIG. 5A, the second period image candidate determined to be appropriate is selected as the second period image by the pair image selection means 106.

When the second period image can be selected, as shown in FIG. 2, a search area is created (Step 300) and then the images are collated (Step 400). It is possible to collate the first period image and the second period image as they are without creating a search area, but the fixed point camera 101 may be shaken by the wind, and in this case, the peripheral edges of the image are inevitably collated (matching). It becomes difficult to do. Therefore, the search area excluding the peripheral edge of the image is created. Specifically, the search area creating means 107 uses the first period image (which may be the second period image) as the search image, and the second period image (which may be the first period image) as the searched image. As shown in FIG. 6A, a search area excluding the peripheral edge of the search image is created. The peripheral edge of the image to be deleted to create the search area can be appropriately designed, for example, to be 5 to 10% of the total area of the image (or the length of one side of the image).

When the searched image (in this case, the second period image) is set by the search area creating means 107 and the search area (in this case, a part of the first period image) is created, the image matching means 102 uses the search area. By collating the searched image with the searched image, the search area and the searched image (that is, the first period image and the second period image) are aligned. Specifically, as shown in FIG. 6B, while the position of the searched image is fixed and one of the search areas is moved, a value indicating the degree of collation each time (hereinafter referred to as "collation level"). ) Is obtained, and the positions of the first period image and the second period image are determined by the arrangement of the search area having the highest collation level. As the collation level, the correlation coefficient between the search area and the searched image can be exemplified.

When the image collating means 102 aligns the first-stage image and the second-stage image, the candidate region extraction means 103 extracts the change region candidate (Step 500). Specifically, pattern matching is performed between the first period image and the second period image in the aligned state, a partial image (pattern) common to both images is extracted, and the position of the common pattern is obtained. The ones with different values are extracted as change area candidates. Here, the change region candidate is not used as it is as the "change region" because it is necessary to extract the "change region" after removing various noises. That is, the pattern is common but the position is different between the first period image and the second period image, in other words, the region where the common pattern is moved (displaced) in the 2 o'clock period is extracted as a change region candidate. The extracted change region candidate includes a "noise region". For example, in FIG. 7, vegetation relatively close to the fixed-point camera 101 occupies the right side of the image, but this vegetation moves frequently due to wind and the like, and of course such a change is not the target change region. .. Therefore, it is necessary to exclude this as a "noise region" and then extract the target "significant change region".

The noise region includes a region where displacement that cannot be considered as slope fluctuation (hereinafter referred to as "directional noise") and a region where changes can be seen only between images obtained in a short time (hereinafter referred to as "directional noise"). "Time noise"), a region where the degree of displacement is small due to pattern matching accuracy (hereinafter, "displacement noise") can be mentioned. Hereinafter, methods for removing each noise will be described.

Some of the anomalous slopes, such as shallow and deep collapses, landslides, and debris flows, fluctuate in the direction of gravity (that is, downward). Therefore, when monitoring the slope that is the object, at least the change region candidate displaced in the direction against gravity (vertically upward) must be removed as "directional noise". Specifically, the directional noise extraction means 108 obtains the displacement vector (displacement amount and displacement direction) of the change region candidate extracted by the candidate region extraction means 103, and if the displacement direction of the displacement vector is approximately upward, the change is obtained. The region candidate is determined as directional noise (Step 600). In determining that the displacement direction of the displacement vector is generally upward, a reference threshold value (hereinafter referred to as “change direction threshold value”) is set in advance, and the displacement direction of the displacement vector is the change direction threshold value. When it is within the range, it should be judged that it is generally upward. As shown in FIG. 8, for example, the change direction threshold value can be set in a range set with a predetermined width on both sides from the vertical upward direction.

As described above, when the first period and the second period acquired in a too short time are compared, changes that are not originally intended due to the movement of animals and plants and the influence of light may be extracted as change region candidates. Therefore, such changes must be removed as "time noise". The time noise extraction means 109 executes this process. Specifically, the first period image and the short time image are obtained by collating the first period image with one or more short-time images (that is, the images acquired between the first period and the second period). The positions are aligned, pattern matching is performed between the first-stage image and the short-time image in that state, and a region that has moved relatively (a region in which the common pattern is displaced) is extracted as time noise (Step 700). Since the short-time image is an image acquired without a sufficient time (predetermined time) elapsed from the first period, the change generated between the first image and the short-time image can be removed as time noise. Is. It should be noted that the change extracted as a result of comparing the short-time images acquired between the first period and the second period can also be removed as time noise.

Depending on the accuracy of pattern matching, even those that have not actually changed may be extracted as change area candidates. In this case, the displacement amount of the displacement vector of the change region candidate is generally small. Alternatively, it is preferable to remove the change region candidate showing a minute displacement amount even if it actually changes, in the sense that it is easy to grasp the abnormality of the slope. Therefore, it is desirable to remove such a change indicating a small displacement amount as "displacement noise". Specifically, the displacement vector of the change region candidate extracted by the candidate region extraction means 103 is obtained, and when the displacement amount of the displacement vector falls below a predetermined threshold value (hereinafter referred to as “change amount threshold value”), the change region is obtained. Candidates are determined as displacement noise.

When the directional noise is extracted by the directional noise extracting means 108, the time noise is extracted by the time noise extracting means 109, and the displacement noise is further extracted, the directional noise and the time are extracted from the change region candidates extracted by the candidate region extraction means 103. The "change area" is determined by removing noise and displacement noise (Step 800). It is the change area extraction means 104 that executes this process.

By the way, when a large change is seen in the entire slope such as a large-scale collapse or a large amount of debris flow, the first period image and the second period image are completely different images. Therefore, even if the image collation means 102 collates the search area with the searched image, the collation level (for example, the correlation coefficient or the luminance difference) shows an extremely small value, but in that state, the first period image. It is not appropriate to align the position of the second period image with. Therefore, as a result of the image collation means 102 collating the search area with the searched image (that is, the first period image and the second period image), the degree of collation (collation level) is a predetermined threshold value (hereinafter, "correlation threshold value"). If it falls below (), it is advisable to output "warning information" as if an abnormal change has occurred on the slope.

The process of outputting "warning information" by the warning information output means 110 will be described with reference to FIG. FIG. 9 is a flow chart showing a flow of processing in which the warning information output means 110 outputs warning information. As shown in this figure, the warning information output means 110 compares the maximum collation level obtained by collating the search area with the searched image by the image collation means 102 and the luminance difference threshold value (Step 401). Then, when the luminance difference threshold is exceeded, a command is given to proceed to the extraction of change area candidates (Step 500) shown in FIG. 2, and when the luminance difference threshold is exceeded, "warning information" is sent to output means such as a display, a printer, and an audio output device. Command to output (Step 900).

The monitoring system of the present invention can also be used to determine changes in natural slopes, cut slopes, embankment slopes, concrete structures such as concrete dams, landfills and soft ground.

100 Surveillance system 101 Fixed point camera 102 Image collation means 103 Candidate area extraction means 104 Change area extraction means 105 Image storage means 106 Pair image selection means 107 Search area creation means 108 Directional noise extraction means 109 Time noise extraction means 110 Warning information output means

Claims (3)

In a system that monitors changes in an object using images from a fixed-point camera installed at one point.
The fixed-point camera that periodically or intermittently acquires images of the object,
The first period image and the second period image are obtained by collating the first period image acquired by the fixed point camera in the first period with the second period image acquired by the fixed point camera in the second period. Image matching means to align the position of
Candidate area extraction means that performs pattern matching between the aligned first-stage image and the second-stage image, and extracts a region in which a partial image common to both images has moved relatively as a change region candidate. When,
The first period image and the short time are obtained by collating the first period image with one or more short time images acquired by the fixed point camera between the first period and the second period. The position of the image is aligned, and pattern matching is performed between the aligned first-stage image and the short-time image, and the region where the partial image common to both images moves relatively is extracted as time noise. Time noise extraction means and
Among the change area candidates, the change area candidate whose position difference generated between the first period image and the second period image exceeds a predetermined change amount threshold value, and the time from the change area candidate. A change area extraction means for extracting the change area candidate excluding noise as a change area, and a change area extraction means.
A monitoring system characterized by being equipped with. Among the change area candidates, the direction in which the change area candidate in which the change direction generated between the first period image and the second period image is within the range of the predetermined change direction threshold value is extracted as directional noise. Further equipped with noise extraction means,
The change direction threshold value is a range set by a predetermined width from the vertical upward direction.
The change region extraction means extracts the change region from the change region candidate excluding the direction noise.
The monitoring system according to claim 1. One of the first period image and the second period image is used as a search image and the other is used as a searched image, and a search area creating means for cutting out a search area excluding the peripheral end portion of the search image is further provided.
The image collation means aligns the positions of the first period image and the second period image by collating the search area with the searched image.
The monitoring system according to claim 1 or 2, wherein the monitoring system is characterized in that.

Images