JP7043992B2 - Change point detection device, change point detection method and change point detection program - Google Patents

Description

The present invention relates to a technique for detecting a change in a control panel from a photographed image of the control panel.

Examples of the technique for detecting a change in a subject from an image of the subject include a change detection device disclosed in Patent Documents 1 and 2.

The detection technology device of Patent Document 1 acquires a plurality of image data groups having different shooting locations and shooting times for the same subject, and shoots the feature points of each image as feature points by a triangular survey method. Grasp the positional relationship of the device and create 3D point cloud data. Then, the change point of the subject is extracted by comparing the three-dimensional point cloud data obtained from each image data group.

The change detection device of Patent Document 2 attaches an AR (Augmented Reality) marker to the subject and detects the change point of the subject for each frame of the image.

Japanese Unexamined Patent Publication No. 2016-110210 Japanese Unexamined Patent Publication No. 2015-158880 Japanese Unexamined Patent Publication No. 2017-39189

Takafumi Aoki, Koichi Ito, Takuma Shibahara, Kiyoshi Nagashima, "High-precision machine vision based on phase-limited correlation method-Aiming for image sensing technology that overcomes the barrier of pixel resolution-," IEICE Fundamentals Review, Vol.1, No. 1, pp.30-40, July 2007.

The equipment illustrated in the control panel has a monochromatic box shape and lacks features, and parts such as levers and switches attached to the equipment are black or glossy, so they may reflect light during shooting. False detection of changes is likely to occur. Therefore, the change point detection technique based on the three-dimensional point cloud data as in Patent Document 1 is not suitable for detecting the change point of the subject having few feature points.

On the other hand, the detection technique of Patent Document 2 detects the change point of the subject based on the AR marker attached to the subject. However, if the subject is, for example, a device owned by the customer, attaching an AR marker to the subject may not meet the customer's request. In addition, when the AR marker is detachably attached to the device like a magnet sheet, the position and orientation of the AR marker may change during maintenance of the device, which may interfere with the detection of the change point. Further, as described above, since devices such as control panels generally lack feature points, it is difficult to extract feature points similar to AR markers and detect changes. In addition, attaching a small-sized AR marker to a relatively large subject such as a control panel makes the detection accuracy of the change point inferior.

In view of the above circumstances, it is an object of the present invention to detect changes in the appearance of the control panel with high accuracy.

Therefore, one aspect of the present invention is a change point detection device that detects changes before and after the repair work of the control panel, and is obtained from image data obtained by photographing the front surface of the board before and after the repair work. A board front specific part that specifies the contour of the board front surface and extracts a board front image, a projection conversion unit that converts the board front image into a projection conversion result image, and a projection conversion result image before and after the work. Is provided with a difference detection unit that generates a change point detection result image that is colored into a subdivided image having a difference between before and after the work.

In one aspect of the present invention, in the change point detection device, the board front surface specifying unit extracts a region specified based on the board front angle coordinates designated for the image data as the board front image.

In one aspect of the present invention, in the change point detection device, the board front specifying portion extracts a region specified based on the board front angle coordinates designated for the image data as the board front image, and at the same time, The board front image is extracted by obtaining matching template images of the board front panel four corners from the board front angle coordinates and specifying the board front after work by the matching template images of the board front board four corners.

In one aspect of the present invention, in the change point detection device, the front surface specifying portion of the board uses a quadrangle having the largest area among the quadrangles included in the image data as the contour of the front surface of the board.

In one aspect of the present invention, in the change detection device, the difference detection unit detects the difference based on the RGB threshold value.

In one aspect of the present invention, in the change detection device, the difference detection unit detects the difference based on the HSV threshold value.

One aspect of the present invention is that in the change detection device, the difference detection unit performs a post-work projection conversion result image by template matching of a hierarchical reduced image of the projection conversion result image before and after the work. A change point detection result image in which the coordinate groups of dissimilar feature points of the projective transformation result image detected in the above work is colored is generated.

In one aspect of the present invention, in the change point detection device, the difference detection unit considers a region of the coordinate group of the dissimilar feature points where the number of adjacent coordinates is equal to or more than a threshold value as a region where the change has occurred.

One aspect of the present invention is a change point detection method in which a computer detects changes before and after the repair work of the control panel, and is obtained from image data obtained by photographing the front surface of the board before and after the repair work. The board front identification process of specifying the contour of the board front surface and extracting the board front image, the projection conversion process of converting the board front image into a projection conversion result image, and the projection conversion result image before and after the work. It has a difference detection process of generating a change point detection result image colored in a subdivided image having a difference between before and after the work.

In one aspect of the present invention, in the change point detection method, the board front surface specifying process extracts a region specified based on the board front surface angle coordinates designated for the image data as the board front image.

In one aspect of the present invention, in the change point detection method, the board front surface specifying process extracts a region specified based on the board front surface angle coordinates designated for the image data as the board front image. The board front image is extracted by obtaining matching template images of the board front panel four corners from the board front angle coordinates and specifying the board front after work by the matching template images of the board front board four corners.

In one aspect of the present invention, in the change point detection method, in the board front surface specifying process, the quadrangle having the largest area among the quadrangles included in the image data is used as the contour of the board front surface.

In one aspect of the present invention, in the change detection method, the difference detection process detects the difference based on the RGB threshold value.

In one aspect of the present invention, in the change detection method, the difference detection process detects the difference based on the HSV threshold value.

One aspect of the present invention is the change detection method, in which the difference detection process is performed by the template matching of the hierarchical reduced image of the projection conversion result image before and after the work, and the projection conversion result image after the work. A change point detection result image is generated in which the coordinate groups of the dissimilar feature points of the projective conversion result image detected in the above work are colored.

In one aspect of the present invention, in the change point detection method, the difference detection process considers a region of the group of dissimilar feature points where the number of adjacent coordinates is equal to or more than a threshold value as a region where the change has occurred.

One aspect of the present invention is a change point detection program that causes a computer to function as the change point detection device, or a change point detection program that causes a computer to execute the change point detection method.

According to the above invention, changes in the appearance of the control panel can be detected with high accuracy.

The block block diagram which showed the outline of the board change point detection system provided with the change point detection apparatus which is one aspect of this invention. It is explanatory drawing of the operation example of the change detection apparatus of FIG. It is explanatory drawing of the image processing process when the board change point detection system of FIG. 1 is used. The explanatory view of the board front surface specifying process in Embodiment 1 of this invention. The explanatory view of the difference detection process in Embodiment 1. FIG. The explanatory view of the pixel comparison process in Embodiment 1. FIG. It is explanatory drawing of the board front surface specifying process by the board front surface specifying part 000 of Embodiment 2 of this invention. The explanatory view of the board front surface specifying process by the board front surface specifying part 010 of Embodiment 2. FIG. The explanatory view of the processing process of the image to be processed in Embodiment 2. The explanatory view of the board front surface specifying process in Embodiment 3. The explanatory view of the processing process of the image to be processed in Embodiment 3. The explanatory view of the extraction process of the board front image based on the board front angle coordinates in Embodiment 3. FIG. The explanatory view of the difference detection process in Embodiment 4. The explanatory view of the pixel comparison process in Embodiment 4. The explanatory view of the difference detection process in Embodiment 5. The explanatory diagram of the hierarchical feature point search process in Embodiment 5. The explanatory view of the coordinate range making process in Embodiment 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Embodiment 1]
(Overview)
When repairing the control panel, unintentional minor changes such as ON / OFF of the switch of the control panel may lead to accidents and damages, so are the changes in the control panel after the repair work appropriate? The importance of confirmation is increasing.

The panel change point detection system 1 of the present embodiment illustrated in FIG. 1 easily and highly accurately grasps the changes in the control panel based on the images of the control panel before and after the repair work.

(Example of mode of board change detection system 1)
The board change point detection system 1 includes a photographing device 20, an input device 30, an image data storage device 40, a change point detection device 00, and a display device 50.

The photographing device 20, the input device 30, the image data storage device 40, the change point detection device 00, and the display device 50 are mounted on the board change point detection system 1 in cooperation with the hardware resources and software resources of the computer.

The photographing device 20 takes a photograph of the control panel before and after the repair work. As the photographing apparatus 20, a general digital camera 21 is applied. For example, when the board change detection system 1 is mounted on a smart terminal such as a smartphone, a tablet, or a notebook computer, the digital camera 21 is built in the smart terminal.

The input device 30 accepts the designation of the panel front angle with respect to the image of the control panel obtained by the shooting by the input operation of the user (for example, the manager of the control panel). The input device 30 explicitly accepts the designation via the input unit 31. As the input unit 31, for example, when the board change point detection system 1 is mounted on the smart terminal, the mode of the user interface that accepts the designation by an input device or a touch operation while being displayed on the display of the smart terminal is provided. Can be mentioned.

The image data storage device 40 stores image data obtained by photographing the control panel.

The change point detection device 00 processes the image data to detect the change point of the image subject.

The display device 50 displays the change point detection result of the image-receiving board via the display unit 500. Examples of the display unit 500 include the display of the smart terminal.

In addition, the panel change point detection system 1 cooperates with the hardware resource and the software resource of the smart terminal to provide a photographing device 20, an input device 30, an image data storage device 40, a change point detection device 00, and a display device 50. It is also possible to configure with a single device.

(Configuration example and operation example of change detection device 00)
As shown in FIG. 1, the change point detecting device 00 includes a panel front surface specifying unit 000,010, a projective conversion unit 001,011 and a difference detecting unit 020.

The board front surface specifying unit 000 identifies the contour of the board front surface from the image data 100 of the board front image obtained by photographing the board front surface before the work of repairing the control panel, and extracts the board front image 101.

The board front surface specifying unit 010 identifies the contour of the board front surface from the image data 110 of the board front image obtained by photographing the board front surface after the repair work, and extracts the board front image 111.

The projective transformation unit 001 converts the board front image 101 into a projective conversion result image 102 by projective transformation based on the coordinates of the corners of the board front.

The projective transformation unit 011 converts the board front image 111 into a projective conversion result image 112 by projective transformation based on the coordinates of the corners of the board front.

The difference detection unit 020 subdivides the projection conversion result image 102 before the work and the projection conversion result image 112 after the work, and colors the subdivided image having a difference between before and after the work. Image 120 is generated.

The flow of image data processing by the change detection device 00 will be described with reference to FIGS. 2 and 3.

First, before the repair work of the control panel, the control panel for which the change point is detected is photographed, and the image data 100 is obtained by the imaging device 20. The panel front angle coordinate group C101 is designated for the image data 100 by the operation of the user of the input device 30. When the image data 100 and the board front angle coordinate group C101 are input to the board front specific unit 000, the board front image 101 is obtained. The board front image 101 and the board front angle coordinate group C101 are converted into a projection conversion result image 102 by the projection conversion unit 001. The projection conversion result image 102 is input to the difference detection unit 020.

As described above, the input image data is converted into an image taken from the front of the control panel. As described below, the above-mentioned panel front surface specifying process and projective conversion process are performed on image data having different shooting locations and shooting times for the same shooting target.

Next, after the control panel is repaired, the control panel is photographed again, and the image data 110 is obtained by the imaging device 20. Similarly to the above, the panel front angle coordinate group C111 is designated for the image data 110 by the operation of the input device 30 by the user. The image data 110 and the board front angle coordinate group C111 are input to the board front specifying unit 010, and the board front image 111 is obtained. The board front image 111 and the board front angle coordinate group C111 are converted into a projection conversion result image 112 by the projection conversion unit 011. The projection conversion result image 112 is input to the difference detection unit 020.

Then, the difference detection unit 020 detects the change point by comparing the image data after the projection conversion obtained for each image data, and obtains the change point detection result image. That is, the difference detection unit 020 compares the projection conversion result image 102 and the projection conversion result image 112 to detect different points, colors the changes on the projection conversion result image 112, and displays the change point detection result image 120. obtain.

By the above processing, even if the two images have different positions of the cameras for shooting with respect to the object to be shot, the changed points can be detected.

(Operation example of board front specific part 000,010)
A specific operation example of the panel front surface specifying portions 000 and 010 will be described with reference to FIG.

The board front specifying units 000 and 010 of the present embodiment execute the following board front specifying process.

S101: The front angle of the board with respect to the input image (image of the board) is specified by the instruction received from the input device 30 operated by the user.

S102: Recognizes the coordinates of the designated board front angle.

In this process, the board front surface specifying unit 000 obtains the board front angle coordinate group C101 by explicitly designating the board front angle in the image data 100 from the user's input device 30.

On the other hand, the board front surface specifying unit 010 obtains the board front angle coordinate group C111 by explicitly designating the board front angle in the image data 110 from the user's input device 30.

S103: The designated corners are connected by a line, and the contour line on the front surface of the board is drawn.

In this process, the contour line drawing function 200 of the board front surface specifying portion 000 draws the board front contour line L100 by connecting the corners of the designated board with a line.

On the other hand, the contour line drawing function 200 of the board front surface specifying portion 010 draws the board front contour line L110 by connecting the corners of the designated board with a line.

S104: The image of the portion surrounded by the contour line on the front surface of the board is cut out.

In this process, the image extraction function 201 of the board front specifying unit 000 extracts only the portion of the input image surrounded by the board front contour line L100 to create the board front image 101. On the other hand, the image extraction function 201 of the board front specifying unit 010 extracts only the portion of the input image surrounded by the board front contour line L110 to create the board front image 111.

S105: The cut-out image of the front surface of the board and the angular coordinates of the front surface of the board are input to the projection conversion unit 001, 011.

In this process, the board front image 101 and the board front angle coordinate group C101 are input to the projection conversion unit 001. On the other hand, the board front image 111 and the board front angle coordinate group C111 are input to the projection conversion unit 011.

S106: The above processes of S101 to S105 are performed on each image before and after the work.

(Operation example of projective conversion unit 001, 011)
A specific operation example of the projective transformation unit 001, 011 will be described with reference to FIG.

The projective transformation unit 001,011 executes the following projective transformation process.

S111: The board front image and the board front angle coordinates are received from the board front specific units 000 and 010.

In this process, the projective conversion unit 001 receives the board front image 101 and the board front angle coordinate group C101 from the board front specific unit 000.

On the other hand, the projective conversion unit 011 receives the board front image 111 and the board front angle coordinate group C111 from the board front specific unit 010.

S112: The image on the front surface of the board is projected and converted into an image taken from the front of the board.

In this process, the projective transformation unit 001 converts the board front image 101 into an image taken from the front of the board by a general method similar to the projective transformation based on the board front angle coordinate group C101, and the projection conversion result image 102. To get.

On the other hand, the projective transformation unit 011 converts the board front image 111 into an image taken from the front of the board by a general method similar to the projective transformation based on the board front angle coordinate group C111, and obtains the projection conversion result image 112. ..

S113: The projection conversion result is input to the difference detection unit 020.

In this process, the projection conversion result image 102 and the projection conversion result image 112 are input to the difference detection unit 020.

S114: The above is performed for the images before and after the work (board front images 101 and 111).

In the illustrated example, the image format, image size, image resolution, area of the front surface of the board drawn in the projection conversion result image, and the position of the front surface of the board in the image of the projection conversion result image 102 or the projection conversion result image 112 are input. It is constant regardless of the image on the front of the board.

(Operation example of difference detection unit 020)
A specific operation example of the difference detection unit 020 will be described with reference to FIG.

The process of the difference detection unit 020 executes the following difference detection process.

S121: Receives the projection conversion result images 102 and 112 before and after the work.

In this process, the projection conversion result image 102 before the work is received from the projection conversion unit 001. In addition, the projection conversion result image 112 after the work is received from the projection conversion unit 011.

S122: Each image is divided into meshes and subdivided, and the coordinates of each subdivided image are stored.

In this process, the image subdivision function 300 of the difference detection unit 020 first subdivides the projection conversion result image 102 obtained from the projection conversion unit 001, and each subdivision in the subdivision image group 103 and the projection conversion result image 102. The coordinate group C103 of the image is obtained.

On the other hand, the image subdivision function 301 of the difference detection unit 020 first subdivides the projection conversion result image 112 obtained from the projection conversion unit 011 and then subdivides each of the subdivision image group 113 and the projection conversion result image 112. The coordinate group C113 of the image is obtained.

At this time, all the subdivided images in the subdivided image group 103 and the subdivided image group 113 have the same size and the same resolution.

S123: In the subdivided images with the same coordinates, the comparison is performed pixel by pixel in the RGB color space.

In this process, the subdivided image comparison function 302 of the difference detection unit 020 compares the subdivided images at the same coordinates on a pixel-by-pixel basis, and the number of pixels is equal to or greater than the pixel number threshold S1 set in advance in the change point detection device 00. If there is a difference, it is regarded as a changed part.

S124: When a pixel having a difference of the RGB threshold value or more exists in the pixel number threshold value or more, the subdivided image is recognized as a difference portion and its coordinates are recorded.

In this process, refer to the RGB color space for pixel-by-pixel comparison. If there is a difference of RGB threshold group S2 or more set in advance in the change point detection device 00 for each pixel in the subdivided image having the same coordinates in the subdivided image group 103 and the subdivided image group 113, the pixel is the pixel. Recognize that there has been a change. The RGB threshold group S2 has thresholds S2R, S2G, and S2B for the difference between each RGB value with respect to R (red), G (green), and B (blue). Even one of the RGB values of each pixel in the subdivided image of the subdivided image group 113 is ± S2R, ± S2G with respect to the RGB value of each pixel in the subdivided image of the subdivided image group 103 having the same coordinates. If it is out of the range of ± S2B, it is considered that the pixel has been changed.

If the number of pixels having a difference in each subdivided image does not reach the pixel number threshold value S1, the pixel comparison in the subdivided image is continued. If the number of pixels with a difference is less than the pixel number threshold value S1 even when all the pixels are compared, it is considered that there is no change in the subdivided image, the comparison of the subdivided images is completed, and the next subdivided image is completed. Make a comparison.

Further, in the above comparison process, when the difference in the number of pixels having the pixel number threshold value S1 or more is confirmed with respect to the pixels in the subdivided image, it is regarded as a changed part as illustrated in FIG. The comparison of the subdivided images is completed. After that, the coordinates of the subdivided image are recorded, and the next subdivided image is compared.

S125: The above processing is performed on all the subdivided images.

In this process, the above comparison process is repeated, all the subdivided images of the subdivided image group 103 and the subdivided image group 113 are compared, and the coordinates of the subdivided images for which the changed points are confirmed are recorded as the changed point coordinate group C120. ..

S126: Only the portion of the subdivided image having a difference is colored and output to the projection conversion result image 112 after the work.

In this process, for the projective transformation result image 112, a region having the same size as the subdivided image is colored at each coordinate of the change point coordinate group C120. As a result, the change detection result image 120 is obtained.

(Effect of Embodiment 1)
The conventional technique such as Patent Document 1 is a method of detecting a change point by using a three-dimensional shape restoration technique. However, in general, the board has few feature points, and the accuracy when performing three-dimensional shape restoration is low. When the technique described in this document was used for the board, the feature points on the board could not be captured during the three-dimensional shape restoration, and the three-dimensional shape restoration could not be performed, or the three-dimensional shape restoration of the board could be performed. However, the accuracy is low, and there are cases where the unchanged part is recognized as a change point.

Further, a conventional technique such as Patent Document 2 is a method of attaching an AR marker to an imaged object and detecting a change point of an object to be photographed. If the AR marker is retrofitted to the board, the position and orientation of the AR marker may change during the modification or repair work of the board. In that case, the positions of all the components such as switches and lamps on the board with respect to the position of the AR marker are recognized as moved, and the changed points cannot be detected. Also, even if the AR marker does not move during the work of modifying or repairing the panel, it is generally accurate to install a small AR marker on an object of a size such as a control panel to recognize the positional relationship. As a result, the parts that have not been changed may be recognized as changes.

On the other hand, the board change point detection system 1 of the present embodiment recognizes the board front surface by the board front surface specifying units 000 and 010, and generates an image of the board front surface taken from the front by the projection conversion unit 001, 011. The difference detection unit 020 determines the change on the front surface of the board based on the RGB value difference. That is, when comparing the front image of the board before and after the work, the color of the same coordinates on the front of the board based on the front angle of the board is used instead of the comparison based on the feature points and AR markers on the front of the board. Compare. This makes it possible to recognize the front surface of the board with higher accuracy and to recognize the changed points with higher accuracy.

As described above, according to the panel change point detection system 1 of the present embodiment, the control panel, which is generally considered to have few characteristic points, is subject to imaging, and the modification is performed under different imaging conditions and lighting conditions. Changes in the board and the board to be repaired can be detected with high accuracy. In addition, it is possible to detect the changed points with high accuracy without installing a special illustration such as an AR marker.

[Embodiment 2]
(Overview)
In the first embodiment, when the angular coordinates of the front surface of the board are set in the image data 100 and the image data 110, the user explicitly sets them in the input unit 31 by operating the input device 30.

On the other hand, in the second embodiment, the setting of the panel front angle coordinates of the image data 110 is automated by utilizing the image processing by the hardware resource of the computer on which the change point detection device 00 is mounted.

In the board change point detection system 1 of the second embodiment, the board change point detection system 1 of the change point detection device 00 is changed from the processing function of FIG. 4 to the processing function of FIG. The same embodiment as that of the first embodiment is used except that the processing function of FIG. 4 is changed to the processing function of FIG.

(Operation example of board front specific part 000,010)
The flow of image processing of the present embodiment will be described with reference to FIGS. 7 to 9.

The board front specifying units 000 and 010 of the second embodiment execute the following board front specifying process.

S201: The user specifies the front angle of the board with respect to the input image of the board before the work.

In this process, as shown in FIG. 7, in the board front surface specifying unit 000, first, the board front angle coordinate group C101 of the input image data 100 is set in the input unit 31 by the operation of the user's input device 30. Explicitly specified.

S202: Recognizes the coordinates of the front corner of the board.

S203: Connect the specified corners with a line and draw the outline of the front of the board.

In this process, the contour line drawing function 210 of the board front surface specifying unit 000 draws the board front contour line L101 by connecting the designated board front angle coordinate group C101 with a line.

S204: The image of the part surrounded by the contour line on the front surface of the board is cut out.

In this process, the image extraction function 211 of the board front specifying unit 000 extracts only the portion of the input image surrounded by the board front contour line L101 to obtain the board front image 101. This board front image 101 is input to the projection conversion unit 001 together with the board front angle coordinate group C101.

S205: The image around the corners of the front surface of the board (four corners of the front board of the board) is cut out and registered as a matching template image.

In this process, the image extraction function 212 of the board front surface specifying unit 000 extracts the board front angle coordinate group C101 peripheral image from the board front image 101 to obtain the matching template image group 104. When the board front angle coordinate group C101 peripheral image is extracted from the board front image 101, it is surrounded by the board front contour line and the image edge on the premise that the board front contour line around the board front corner is included. By extracting the area, the matching template image group 104 obtained by extracting the periphery of the front corner of the board from the front of the board is obtained. The matching template image group 104 is input to the board front specific unit 010.

S206: Template matching is performed on the input image of the board after the work.

In this process, as shown in FIG. 8, the template matching function 213 of the board front specifying unit 010 first uses the matching template image group 104 as a matching template for the input image data 110, and performs template matching. conduct. From the obtained matching results, the image portion closest to each matching template image on the image data 110 is selected, and the corner end of the image portion is designated as the board front angle coordinate group C111.

S207: Recognizes the coordinates of the front surface angle of the board.

S208: The detected corners of the board are connected by a line, and the outline of the front surface of the board is drawn.

As in the case of FIG. 9, the matching template image group 104, that is, which of the four matching template images is matched against, which corner end is used as the board front angle coordinate is different. .. Therefore, in this process, the contour line drawing function 214 of the board front surface specifying portion 010 draws the board front contour line L111 by connecting the board front angle coordinate group C111 with a line.

S209: The image of the part surrounded by the contour line on the front surface of the board is cut out.

In this process, the drawing extraction function 215 of the board front specific portion 010 extracts only the portion of the input image surrounded by the board front contour line L111 to obtain the board front image 111.

S210: The clipped board front image and the angle coordinates of the board front are input to the projection conversion unit.

In this process, the board front image 111 and the board front angle coordinate group C111 are input to the projection conversion unit 011.

As described above, according to the board change point detection system 1 of the present embodiment, as in the first embodiment, the board front image 101 and the board front angle coordinate group C101 obtained above are connected to the projective conversion unit 001 on the board front surface. The image 111 and the board front angle coordinate group C111 are input to the projective conversion unit 011. Then, the projection conversion result image 102 and the projection conversion result image 112 obtained by the projection conversion unit 011 are input to the difference detection unit 020, and the change point detection result image 120 is obtained.

In particular, the present embodiment is not limited to the above-mentioned template matching method, and can be realized by applying, for example, the arrangement detection technique of Patent Document 3.

As described above, according to the board change point detection system 1 of the present embodiment, the setting of the board front angle coordinates in the image data 110 is automated by utilizing the image processing of the computer, and thus the invention of the first embodiment. In addition to the effect of, the efficiency of data processing can be improved.

[Embodiment 3]
(Overview)
In the above-described first embodiment, when the angular coordinates of the front surface of the board are specified in the image data 100 and the image data 110, the user explicitly sets the change point detection device 00.

Further, in the second embodiment, if the user explicitly sets the board front angle coordinates of the image data 100, the board front angle coordinates of the image data 110 are automatically recognized by the change point detection device 00.

On the other hand, in the third embodiment of FIG. 11, it is assumed that the front surface of the board is the largest in the photograph taken, and the user does not set the front angle coordinates of the board, and any part of the front surface of the board. Even if the above is modified or repaired, the change detection device 00 automatically recognizes the front angle of the panel.

The panel change point detection system 1 of the third embodiment does not include the input device 30 of the first embodiment, and the processing contents of the panel front surface specifying unit 000 and the panel front surface specifying unit 010 of the change point detection device 00 are replaced with those in FIG. It is the same aspect of Embodiment 1 except that it is the content of FIG.

(Operation example of board front specific part 000,010)
The flow of image processing of the present embodiment will be described with reference to FIGS. 10 to 12.

The board front specifying units 000 and 010 of the third embodiment execute the following board front specifying process.

S301: Straight line detection and angle detection are performed on the input board image.

In this process, the straight line detection function 220 of the board front surface specifying unit 000 first extracts the linear component expression group E100 existing in the input image data 100 by using a general method similar to the Hough transform. On the other hand, the straight line detection function 220 of the board front surface specifying unit 010 first extracts the linear component expression group E110 existing in the input image data 110 by using a general method similar to the Hough transform.

Further, the corner detection function 221 of the panel front surface specifying unit 000 detects the corners existing in the image data 100 by using a general method similar to the corner detection, and obtains the corner coordinate group C100. On the other hand, the corner detection function 221 of the panel front surface specifying unit 010 detects the corners existing in the image data 110 by using a general method similar to the corner detection, and obtains the corner coordinate group C110.

S302: If the detected angular coordinates exist on the detected straight line, an image in which a line is drawn only between the coordinates is created.

In this process, the all-straight line contour drawing function 222 of the board front specific portion 000 pays attention to each straight line component formula of the straight line component formula group E100, and when a plurality of angular coordinates exist on the line of the straight line component formula, the angle thereof. A straight line is drawn between the coordinates (Fig. 12). As a result, a straight line contour image 105 in which only the full straight line contour existing in the image data 100 is drawn can be obtained.

On the other hand, the all-straight line contour drawing function 222 of the board front specific portion 010 pays attention to each straight line component expression of the straight line component expression group E110, and when a plurality of angular coordinates exist on the line of the linear component expression, between the angular coordinates. Draw a straight line in (Fig. 12). As a result, a straight line contour image 115 in which only the full straight line contour existing in the image data 110 is drawn can be obtained.

S303: The area of all the quadrangles surrounded by the drawn lines is calculated, and an image in which only the quadrangle having the largest area is drawn is created.

In this process, the maximum area plane selection function 223 of the board front surface specifying portion 000 calculates the area of all the quadrangles of the linear contour image 105 and selects the quadrangle which is the maximum area plane. As a result, the maximum area plane contour image 106 obtained by drawing only the maximum area plane of the straight line contour image 105 can be obtained.

On the other hand, the maximum area plane selection function 223 of the board front specific portion 010 calculates the area of all the quadrangles of the linear contour image 115 and selects the quadrangle which is the maximum area plane. As a result, the maximum area plane contour image 116 in which only the maximum area plane of the straight line contour image 115 is drawn can be obtained.

S304: The image in which the quadrangle with the maximum area is drawn is superimposed on the original input image, and the image is extracted only within the range surrounded by the quadrangle. As a result, the image is obtained by extracting only the front surface of the board.

In this process, the image extraction function 224 of the board front surface specifying portion 000 superimposes the image data 100 on the maximum area plane contour image 106 and cuts out only the image portion within the maximum area plane contour. As a result, the board front image 101 is obtained.

On the other hand, the image extraction function 224 of the board front surface specifying portion 010 superimposes the image data 110 on the maximum area plane contour image 116 and cuts out only the image portion in the maximum area plane contour. As a result, the board front image 111 is obtained.

S305: The angular coordinates are superimposed on the image in which the quadrangle with the maximum area is drawn, and the angular coordinates of the quadrangle with the maximum area are selected. As a result, the angular coordinates on the front surface of the board are recognized.

In this process, the board front angle coordinate selection function 225 of the board front specifying unit 000 superimposes the angle coordinate group C100 on the maximum area plane contour image 106, and selects the angle coordinates of the maximum area plane from the angle coordinate group C100. As a result, the board front angle coordinate C101 is obtained.

On the other hand, the board front angle coordinate selection function 225 of the board front specific portion 010 superimposes the angle coordinate group C110 on the maximum area plane contour image 116 and selects the angle coordinates of the maximum area plane from the angle coordinate group C100. As a result, the board front angle coordinate C111 is obtained.

S306: The cut out board front image and the angular coordinates of the board front are input to the projection conversion unit.

In this process, the board front image 101 and the board front angle coordinate group C101 obtained in S305 are input to the projection conversion unit 001. On the other hand, the board front image 111 and the board front angle coordinate group C111 obtained in S305 are input to the projection conversion unit 011.

S307: The above processes of S301 to S306 are performed on each image before and after the work.

As described above, according to the board change point detection system 1 of the present embodiment, as in the first embodiment, the board front image 101 and the board front angle coordinate group C101 obtained above are connected to the projective conversion unit 001 on the board front surface. The image 111 and the board front angle coordinate group C111 are input to the projective conversion unit 011. Then, the projection conversion result image 102 and the projection conversion result image 112 obtained by the projection conversion unit 011 are input to the difference detection unit 020, and the change point detection result image 120 is obtained.

Therefore, according to the board change point detection system 1 of the present embodiment, in addition to the effects of the inventions of the first and second embodiments, the user does not have to set the front angle coordinates of the board, and the speed of on-site work can be improved. It is planned. Further, even if the front corner of the board is changed in the modification and repair work, the front corner of the board can be recognized.

[Embodiment 4]
(Overview)
The difference detection unit 020 of the fourth embodiment shown in FIG. 13 refers to the HSV color space, not the RGB color space, when comparing the subdivided image group 103 and the subdivided image group 113. In the HSV color space, hue (H), saturation (S), and lightness (V) are treated as individual parameters.

The board change point detection system 1 of the fourth embodiment has the same aspect as that of the first embodiment except that the processing content of the difference detection unit 020 of the change point detection device 00 is the content of FIG. 13 instead of FIG. Is.

(Operation example of difference detection unit 020)
The flow of difference detection of the present embodiment will be described with reference to FIGS. 13 and 14.

The difference detection unit 020 of the fourth embodiment executes the following difference detection process.

S401: The projection conversion result image 102 before the work is received from the projection conversion unit 001. On the other hand, the projection conversion result image 112 after the work is received from the projection conversion unit 011.

S402: Each image is divided into meshes and subdivided, and the coordinates of each subdivided image are stored.

In this process, the difference detection unit 020 first subdivides the projection conversion result image 102 obtained from the projection conversion unit 001, and divides the subdivided image group 103 and the coordinate group C103 of each subdivided image of the projection conversion result image 102. obtain. On the other hand, the projection conversion result image 112 obtained from the projection conversion unit 011 is subdivided, and the subdivided image group 113 and the coordinate group C113 of each subdivided image of the projection conversion result image 112 are obtained.

S403: In the subdivided image with the same coordinates, the hues are compared pixel by pixel in the HSV color space.

In this process, all the subdivided images of the subdivided image group 103 and the subdivided image group 113 have the same size and the same resolution. The subdivided images at the same coordinates are compared on a pixel-by-pixel basis, and if there is a difference in the number of pixels of the pixel number threshold value S1 or more set in advance in the change point detection device 00, processing is performed to consider that there is a change.

Refer to the hue in the HSV color space for pixel-by-pixel comparison. Whether or not each pixel in the subdivided image having the same coordinates in the subdivided image group 103 and the subdivided image group 113 is changed based on the hue threshold value S3 set in advance on the change point detection device 00. Make a judgment.

The hue threshold value S3 in the fourth embodiment indicates a hue difference threshold value (HSV threshold value) in the HSV color space. The HSV color space is a color space composed of three elements of hue (H), saturation (S), and lightness (V), and is characterized in that hue is expressed by one parameter. As a result, even if the brightness of the image to be photographed before and after the work is different, it is possible to compare only the hue of the image to be photographed by comparing only the hue H value.

When two colors with the same hue but different brightness are expressed by RGB values in the RGB color space, the three parameters of R value, G value, and B value are different even though they have the same hue. Therefore, when the brightness of the image to be photographed before and after the work is different, it is difficult to determine the hue difference in the RGB color space as in the first embodiment, and if the threshold value is set incorrectly, it depends on the brightness of the image to be photographed. May mistakenly recognize the unchanged part as a change.

In the fourth embodiment, by comparing only the hue H value in the HSV color space, it is easier to set the threshold value when determining the hue difference even if the brightness of the image to be photographed before and after the work is different. As a result, erroneous recognition due to erroneous threshold setting is unlikely to occur.

S404: When a pixel having a difference of the hue threshold value or more exists in the pixel number threshold value or more, the subdivided image is recognized as a difference portion and its coordinates are recorded.

In this process, the hue value of each pixel in the subdivided image of the subdivided image group 103 is H1, and the hue value of each pixel in the subdivided image of the subdivided image group 113 having the same coordinates is H2. If the smaller of -H2 | and (360- | H1-H2 |) is S3 or higher, it is considered that the pixel has been changed. If the number of pixels having a difference in each subdivided image does not reach the pixel number threshold value S1, the pixel comparison in the subdivided image is continued. If the number of pixels with a difference is less than the pixel number threshold value S1 even when all the pixels are compared, it is considered that there is no change in the subdivided image, the comparison of the subdivided images is completed, and the next subdivided image is completed. Make a comparison. Further, in the above comparison process, when a difference in the number of pixels having a pixel number threshold value S1 or more is confirmed with respect to the pixels in the subdivided image, it is regarded as a changed portion, and the comparison of the subdivided images is terminated at that point. (Fig. 14). After that, the coordinates of the subdivided image are recorded, and the next subdivided image is compared.

S405: The above processing is performed on all the subdivided images.

The comparison processing of S403 and S404 is repeated, all the subdivided images of the subdivided image group 103 and the subdivided image group 113 are compared, and the coordinates of the subdivided image confirmed to be changed are recorded as the change point coordinate group C120.

S406: Only the portion of the subdivided image having a difference is colored and output to the projection conversion result image after the work.

In this process, for the projective transformation result image 112, a region having the same size as the subdivided image is colored at each coordinate of the change point coordinate group C120. As a result, the change point detection result image 120 is obtained as shown in FIG.

If the brightness of the image to be photographed differs before and after the work, it may be recognized as a change even in a place where the change has not been made. This is because when two colors with the same hue but different brightness are expressed by RGB values, the three parameters of R value, G value, and B value may all be different. For this reason, if the RGB threshold group S2 is set incorrectly, there is a possibility that the unchanged part will be mistakenly recognized as a changed point depending on the brightness of the image subject.

On the other hand, according to the board change point detection system 1 of the present embodiment, by setting a threshold value only for the hue (H), the subdivided image group 103 and the subdivided image group 113 can be compared only by the hue. It will be easy. Therefore, according to the present embodiment, in addition to the effects of the inventions of the first to third embodiments, change point recognition can be performed with high accuracy even when the brightness of both image groups is different.

[Embodiment 5]
(Overview)
In the first and fourth embodiments, the projection conversion result image 102 and the projection conversion result image 112 are divided into a subdivided image group 103 and a subdivided image group 113, and the colors are compared in the subdivided images having the same coordinates. Is detected.

On the other hand, when the difference detection unit 020 of the present embodiment shown in FIG. 15 compares the projection conversion result image 102 and the projection conversion result image 112, the difference detection unit 020 does not compare the colors of the same coordinates of both. The feature points of the projective transformation result image 102 are found from the projective transformation result image 112. As a result, even if the position coordinates of the board surface components included in the projection conversion result image 102 and the projection conversion result image 112 are locally different, change point recognition can be performed with high accuracy.

(Operation example of difference detection unit 020)
The flow of difference detection of the present embodiment will be described with reference to FIGS. 15 to 17.

The difference detection unit 020 of the fifth embodiment executes the following difference detection process.

S501: The projection conversion result image 102 before the work is received from the projection conversion unit 001. On the other hand, the projection conversion result image 112 after the work is received from the projection conversion unit 011.

S502: The reduced images 102a and 112a whose image size is reduced with respect to the projection conversion result images 102 and 112 before and after the work are created.

In this process, the image reduction function 320 of the difference detection unit 020 reduces the image size of the projection conversion result image 102 obtained from the projection conversion unit 001 to obtain a reduced image 102a. On the other hand, the image reduction function 321 of the difference detection unit 020 reduces the image size of the projection conversion result image 112 obtained from the projection conversion unit 011 to obtain a reduced image 112a.

S503: Reduced images 102b and 112b whose image size is further reduced with respect to the reduced images 102a and 112a are created.

In this process, the image reduction function 322 of the difference detection unit 020 further reduces the image size of the reduced image 102a and obtains the reduced image 102b as shown in FIG. On the other hand, the image reduction function 323 of the difference detection unit 020 further reduces the image size of the reduced image 112a and obtains the reduced image 112b as shown in the figure.

S504: The hierarchical feature point search process is performed based on the projection conversion result images before and after the work and the reduced images obtained in the steps of S502 to S503.

In this process, the hierarchical feature point search processing function 324 of the difference detection unit 020 is hierarchical with respect to the projection conversion result image 102, the projection conversion result image 112, the reduced image 102a, the reduced image 112a, the reduced image 102b, and the reduced image 112b. Performs feature point search processing.

In the hierarchical feature point search process, first, template matching is performed between the reduced image 102b and the reduced image 112b having the lowest resolution. The feature points judged to be different in this process are stored. Further, template matching of the reduced image 102a and the reduced image 112a having high resolution is performed for the region determined to have no difference. The feature points judged to be different in this process are stored. Further, template matching of the projective transformation result image 102 and the projective transformation result image 112 is performed for the region determined to have no difference. The feature points determined to be different in this process are stored . As such a hierarchical feature point search process, a general method called hierarchical template matching, hierarchical matching, or the like may be applied (for example, Non-Patent Document 1).

S505: A coordinate group of feature points having a low degree of similarity obtained by the hierarchical feature point search process is obtained.

In this process, the hierarchical feature point search processing function 324 considers that the feature points whose similarity is equal to or higher than the similarity threshold S4 are unchanged, and the coordinate group of the feature points whose similarity is lower than the similarity threshold S4 is not. It is obtained as a coordinate group C130 of similar feature points.

S506: Of the coordinate groups of the change point detection area obtained by the coordinate range conversion process, the area where adjacent coordinates exist is regarded as the change range.

In this process, as shown in FIG. 17, the coordinate range processing function 325 of the difference detection unit 020 is a circle whose radius is the coordinate adjacent distance R1 centered on a certain coordinate in the coordinate group C130 of the dissimilar feature points. Only when there are coordinates of 5 or more adjacent coordinate number thresholds in the area, the area is regarded as the changed area and is regarded as the changed point detection area. In the case of the figure, the regions shown in (1) and (3) are detected as change point detection regions. By performing the above coordinate rangeization process for all the coordinates of the coordinate group C130 of the dissimilar feature points, the change point detection area group A130 can be obtained.

S507: The predetermined change range in the projection conversion result image 112 is colored and output.

In this process, the change point detection area coloring function 326 of the difference detection unit 020 obtains the change point detection result image 120 by coloring the area of the change point detection area group A130 in the projection conversion result image 112.

When the pre-work board image 100 and the post-work board image 110 are shot, if the positional relationship of the shooting device with respect to the board to be imaged is significantly different, the projection conversion result image is obtained even if the image is changed to the image shot from the front of the board by the projection conversion process. The position coordinates of the board components in 102 and the projective conversion result image 112 may be different. This causes a decrease in the change detection accuracy of the change detection system.

On the other hand, when comparing the projection conversion result image 102 and the projection conversion result image 112, the difference detection unit 020 of the present embodiment does not compare the colors of the same coordinates, but projects by the hierarchical feature point search process. The feature points are found from the conversion result image 102 from the projection conversion result image 112.

In particular, according to the hierarchical feature point search process, the search window size is fixed, and template matching is performed in order from the reduced image 102b and the reduced image 112b, which have low resolutions. Therefore, the projection conversion result image 112 to the projection conversion result image 102 The feature points inside are found.

As described above, according to the present embodiment, the positional relationship of the image pickup device with respect to the image to be imaged is different, so that the position coordinates of the board surface components in the projection conversion result image are local as in the first and fourth embodiments. Even if they differ from each other, change point detection can be performed with high accuracy. Further, in the hierarchical feature point search process of the present embodiment, the search range is limited for each layer, and the presence or absence of feature points having a high degree of similarity in the designated search range is used as a determination criterion. Therefore, even compared with the methods 1 and 4 in which the colors of the same coordinates are compared in the projective transformation result image, it is robust against the difference in the local position coordinates of the board component in the projective transformation result image. Become. In particular, it is suitable for searching for feature points on a board in which feature points are lined up in a similar manner. The hierarchical feature point search processing method is not limited, and can be realized by using a general method.

[Example of another aspect of the present invention]
As another example of the present invention, there is a change point detection program in which the computer is used as the above-mentioned change point detection device 00 or the change point detection method is executed by the computer. This program can be provided via a well-known computer-readable recording medium or a network such as the Internet.

The present invention is not limited to the embodiments described above, and can be implemented in various embodiments within the scope of the claims of the present invention.

1 ... Board change point detection system 30 ... Input device 40 ... Image data storage device 00 ... Change point detection device, 000,010 ... Board front specific unit, 001,011 ... Projection conversion unit, 020 ... Difference detection unit 50 ... Display device

Claims (15)

It is a change point detection device that detects changes before and after the work of repairing the control panel.
The work front board that extracts the area specified based on the board front angle coordinates before the work specified for the image data obtained by shooting the board front before the work as the board front image before the work . Front specific part and
After the work , the area specified based on the front angle coordinates of the board after the work specified for the image data obtained by photographing the front of the board after the work of the repair is extracted as the front image of the board after the work . Front specific part and
With the pre- work projection conversion unit that converts the pre-work board front image into a pre-work projection conversion result image by projective conversion based on the pre-work board front angle coordinates .
The post- work projection conversion unit that converts the post-work board front image into a post-work projection conversion result image by projective conversion based on the board front angle coordinates after the work, and the post-work projection conversion result images before and after the work. The change point detection result of the control panel is obtained by detecting and coloring the subdivided images having a difference between before and after the work from the subdivided images before and after the work obtained by subdividing the work. A change detection device characterized by having a difference detection unit that generates an image. The front panel front specifying portion of the work extracts a region specified based on the front angle coordinates of the front panel before the work specified for the image data before the work as a front image of the board before the work, and also extracts the front surface of the board. Obtain matching template images of the four corners of the front panel before the work based on the angular coordinates ,
The post-work board front specific portion extracts the post-work board front image based on the post-work board front contour line obtained by template matching between the matching template image and the post-work board front image data. The change detection device according to claim 1. The front surface specifying portion of the work front identifies the quadrangle having the largest area among the quadrangles included in the image data before the work as the contour of the front surface of the board.
The change point detection according to claim 1 or 2 , wherein the work rear panel front surface specifying portion identifies the quadrangle having the largest area among the quadrangles included in the image data after the work as the contour of the board front surface. Device. The change detection device according to any one of claims 1 to 3 , wherein the difference detection unit detects the difference based on the RGB threshold value of the subdivided image group before and after the work. .. The change detection device according to any one of claims 1 to 3 , wherein the difference detection unit detects the difference based on the HSV threshold value of the subdivided image group before and after the work. .. The difference detection unit is
The first reduced image of the pre-work and post-work projection conversion result images, in which the image size is reduced with respect to the pre-work and post-work projection conversion result images, is created.
A second reduced image of the projection conversion result image before and after the work was created by further reducing the image size with respect to the first reduced image before and after the work.
Template matching of the second reduced image before and after the work is performed, and template matching of the first reduced image before and after the work is further performed for the area determined to have no difference by this template matching. Then, template matching of the projection conversion result image before and after the work is further performed for the area determined to have no difference by this template matching.
A group of dissimilar feature points of the projective conversion result image before and after the work detected by the template matching of the projective conversion result image, and the pre-work and the work detected by the template matching of the first reduced image. A group of dissimilar feature points of the first reduced image after the work, and dissimilar feature points of the second reduced image before and after the work detected by template matching of the second reduced image. Generating a change detection result image with colored groups
The change detection device according to claim 1. The change point detection device according to claim 6 , wherein the difference detection unit regards a region of the group of dissimilar feature points whose number of adjacent coordinates is equal to or greater than a threshold value as a changed region. It is a change detection method in which the computer detects the changes before and after the repair work of the control panel.
The work front board that extracts the area specified based on the board front angle coordinates before the work specified for the image data obtained by shooting the board front before the work as the board front image before the work . Front identification process and
After the work , the area specified based on the front angle coordinates of the board after the work specified for the image data obtained by photographing the front of the board after the work of the repair is extracted as the front image of the board after the work . Front identification process and
The pre- work projection conversion process of converting the pre-work board front image into a pre-work projection conversion result image by projective transformation based on the pre-work board front angle coordinates .
Post- work projection conversion process of converting the post-work board front image into a post-work projection conversion result image by projective conversion based on the board front angle coordinates after the work, and post-work projection conversion result images before and after the work. The change point detection result of the control panel is obtained by detecting and coloring the subdivided images having a difference between before and after the work from the subdivided images before and after the work obtained by subdividing the work. A change detection method having a difference detection process for generating an image. In the process of specifying the front surface of the board before the work, the area specified based on the front angle coordinates of the board before the work specified for the image data before the work is extracted as the front image of the board before the work, and the front surface of the board is specified. Obtain matching template images of the four corners of the front panel before the work based on the angular coordinates ,
In the post-work board front surface specifying process, the post-work board front image is extracted based on the post-work board front contour line obtained by template matching between the matching template image and the post-work board front image data. The change detection method according to claim 8 , wherein the change is detected. In the process of specifying the front surface of the front panel of the work, the quadrangle having the largest area among the quadrangles included in the image data before the work is specified as the contour of the front surface of the board.
The change point detection according to claim 8 or 9 , wherein the process of specifying the front surface of the board after the work identifies the quadrangle having the largest area among the quadrangles included in the image data after the work as the contour of the front surface of the board. Method. The change detection method according to any one of claims 8 to 10 , wherein the difference detection process detects the difference based on the RGB threshold value of the subdivided image group before and after the work. .. The change detection method according to any one of claims 8 to 10 , wherein the difference detection process detects the difference based on the HSV threshold value of the subdivided image group before and after the work. .. The difference detection process is
The first reduced image of the pre-work and post-work projection conversion result images, in which the image size is reduced with respect to the pre-work and post-work projection conversion result images, is created.
A second reduced image of the projection conversion result image before and after the work was created by further reducing the image size with respect to the first reduced image before and after the work.
Template matching of the second reduced image before and after the work is performed, and template matching of the first reduced image before and after the work is further performed for the area determined to have no difference by this template matching. Then, template matching of the projection conversion result image before and after the work is further performed for the area determined to have no difference by this template matching.
A group of dissimilar feature points of the projective conversion result image before and after the work detected by the template matching of the projective conversion result image, and the pre-work and the work detected by the template matching of the first reduced image. A group of dissimilar feature points of the first reduced image after the work, and dissimilar feature points of the second reduced image before and after the work detected by template matching of the second reduced image. Generating a change detection result image with colored groups
8. The change detection method according to claim 8 . The change point detection method according to claim 13 , wherein the difference detection process considers a region of the group of dissimilar feature points whose number of adjacent coordinates is equal to or greater than a threshold value as a changed region. A change point detection program comprising operating a computer as the change point detection device according to any one of claims 1 to 7 .

Images